[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:01:49 dist-test-slave-n106 krb5kdc[26044](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:01:49 dist-test-slave-n106 krb5kdc[26044](info): set up 2 sockets
Sep 24 16:01:49 dist-test-slave-n106 krb5kdc[26044](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 16:01:50.193216 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:01:50 dist-test-slave-n106 krb5kdc[26044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:01:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:01:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
I20260924 16:01:50.319491 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:38529
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:35787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:38529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:01:50.528043 26057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:01:50.528331 26057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:01:50.528373 26057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:01:50.532541 26057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:01:50.532611 26057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:01:50.532631 26057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:01:50.532649 26057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:01:50.532665 26057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:01:50.538110 26057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:35787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:38529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:38529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:01:50.539247 26057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:01:50.540345 26057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:01:50.593088 26057 server_base.cc:1034] running on GCE node
W20260924 16:01:50.593101 26062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:01:50.593058 26065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:01:50.593297 26063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:01:50.594071 26057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:01:50.595484 26057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:01:50.596679 26057 hybrid_clock.cc:648] HybridClock initialized: now 1790265710596669 us; error 33 us; skew 500 ppm
Sep 24 16:01:50 dist-test-slave-n106 krb5kdc[26044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265710, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:01:50.599830 26057 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:01:50.600898 26057 webserver.cc:460] Webserver started at http://127.25.109.62:35905/ using document root <none> and password file <none>
I20260924 16:01:50.601476 26057 fs_manager.cc:359] Metadata directory not provided
I20260924 16:01:50.601567 26057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:01:50.601804 26057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:01:50.604101 26057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "a2655d9419a34f4aa300d239edf3929c"
format_stamp: "Formatted at 2026-09-24 16:01:50 on dist-test-slave-n106"
server_key: "5c9a2e5b15233d376a20ddbcd0f2f7b3"
server_key_iv: "338eb4f9842c013abb808ea7512e9d99"
server_key_version: "encryptionkey@0"
I20260924 16:01:50.604717 26057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "a2655d9419a34f4aa300d239edf3929c"
format_stamp: "Formatted at 2026-09-24 16:01:50 on dist-test-slave-n106"
server_key: "5c9a2e5b15233d376a20ddbcd0f2f7b3"
server_key_iv: "338eb4f9842c013abb808ea7512e9d99"
server_key_version: "encryptionkey@0"
I20260924 16:01:50.608605 26057 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:01:50.611125 26072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:50.612473 26057 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:01:50.612630 26057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "a2655d9419a34f4aa300d239edf3929c"
format_stamp: "Formatted at 2026-09-24 16:01:50 on dist-test-slave-n106"
server_key: "5c9a2e5b15233d376a20ddbcd0f2f7b3"
server_key_iv: "338eb4f9842c013abb808ea7512e9d99"
server_key_version: "encryptionkey@0"
I20260924 16:01:50.612746 26057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:01:50.637384 26057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:01:50.641234 26057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:01:50.641523 26057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:01:50.661136 26057 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:38529
I20260924 16:01:50.661167 26124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:38529 every 8 connection(s)
I20260924 16:01:50.662673 26057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:01:50.668159 26125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:01:50.669551 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 26057
I20260924 16:01:50.669730 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:01:50.671501 26036 external_mini_cluster.cc:1428] Setting key 76b004713f09171d400af796fad8dd99
I20260924 16:01:50.678076 26125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2655d9419a34f4aa300d239edf3929c: Bootstrap starting.
I20260924 16:01:50.681383 26125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2655d9419a34f4aa300d239edf3929c: Neither blocks nor log segments found. Creating new log.
I20260924 16:01:50.682628 26125 log.cc:824] T 00000000000000000000000000000000 P a2655d9419a34f4aa300d239edf3929c: Log is configured to *not* fsync() on all Append() calls
I20260924 16:01:50.684967 26125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2655d9419a34f4aa300d239edf3929c: No bootstrap required, opened a new log
Sep 24 16:01:50 dist-test-slave-n106 krb5kdc[26044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:01:50.693145 26125 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2655d9419a34f4aa300d239edf3929c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2655d9419a34f4aa300d239edf3929c" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 38529 } }
I20260924 16:01:50.693375 26125 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2655d9419a34f4aa300d239edf3929c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:01:50.693536 26125 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2655d9419a34f4aa300d239edf3929c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2655d9419a34f4aa300d239edf3929c, State: Initialized, Role: FOLLOWER
I20260924 16:01:50.694703 26125 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2655d9419a34f4aa300d239edf3929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2655d9419a34f4aa300d239edf3929c" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 38529 } }
I20260924 16:01:50.694845 26125 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2655d9419a34f4aa300d239edf3929c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:01:50.694952 26125 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2655d9419a34f4aa300d239edf3929c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:01:50.695075 26125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2655d9419a34f4aa300d239edf3929c [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:01:50.696053 26125 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2655d9419a34f4aa300d239edf3929c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2655d9419a34f4aa300d239edf3929c" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 38529 } }
I20260924 16:01:50.696480 26125 leader_election.cc:302] T 00000000000000000000000000000000 P a2655d9419a34f4aa300d239edf3929c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2655d9419a34f4aa300d239edf3929c; no voters:
I20260924 16:01:50.696882 26125 leader_election.cc:288] T 00000000000000000000000000000000 P a2655d9419a34f4aa300d239edf3929c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:01:50.697059 26130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2655d9419a34f4aa300d239edf3929c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:01:50.697329 26130 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2655d9419a34f4aa300d239edf3929c [term 1 LEADER]: Becoming Leader. State: Replica: a2655d9419a34f4aa300d239edf3929c, State: Running, Role: LEADER
I20260924 16:01:50.697701 26130 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2655d9419a34f4aa300d239edf3929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2655d9419a34f4aa300d239edf3929c" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 38529 } }
I20260924 16:01:50.697896 26128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:50.676059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57016 (local address 127.25.109.62:38529)
0924 16:01:50.676839 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:50.676858 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:50.677637 (+ 779us) server_negotiation.cc:407] Connection header received
0924 16:01:50.681722 (+ 4085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:50.681747 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:50.682261 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:50.682648 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:50.685216 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:50.686279 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:50.687499 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:50.688024 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:50.688198 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:50.692079 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:50.692084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:50.692100 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:50.692128 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:50.694162 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:50.694730 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:50.694743 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:50.694747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:50.694901 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:50.695328 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:50.695331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:50.695333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:50.695742 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:50.695961 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:50.696665 (+ 704us) server_negotiation.cc:299] Negotiation successful
0924 16:01:50.696935 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":234,"threads_started":1}
I20260924 16:01:50.698702 26125 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2655d9419a34f4aa300d239edf3929c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:01:50.704372 26132 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2655d9419a34f4aa300d239edf3929c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2655d9419a34f4aa300d239edf3929c. Latest consensus state: current_term: 1 leader_uuid: "a2655d9419a34f4aa300d239edf3929c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2655d9419a34f4aa300d239edf3929c" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 38529 } } }
I20260924 16:01:50.704493 26131 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2655d9419a34f4aa300d239edf3929c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2655d9419a34f4aa300d239edf3929c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2655d9419a34f4aa300d239edf3929c" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 38529 } } }
I20260924 16:01:50.704556 26132 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2655d9419a34f4aa300d239edf3929c [sys.catalog]: This master's current role is: LEADER
I20260924 16:01:50.704617 26131 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2655d9419a34f4aa300d239edf3929c [sys.catalog]: This master's current role is: LEADER
W20260924 16:01:50.707360 26144 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a2655d9419a34f4aa300d239edf3929c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:01:50.707422 26144 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:01:50.707501 26146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:01:50.708151 26146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:01:50.713815 26146 catalog_manager.cc:1347] Generated new cluster ID: c3ea6d29ad4248819ada11aad46276a6
I20260924 16:01:50.713877 26146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:01:50.723486 26146 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:01:50.724431 26146 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:01:50.728976 26146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2655d9419a34f4aa300d239edf3929c: Generated new TSK 0
I20260924 16:01:50.729815 26146 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
I20260924 16:01:50.759012 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:0
--local_ip_for_outbound_sockets=127.25.109.1
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:38529
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:35787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:01:50.887715 26151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:01:50.887956 26151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:01:50.888001 26151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:01:50.892104 26151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:01:50.892165 26151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:01:50.892225 26151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:01:50.897699 26151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:35787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:38529
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:01:50.898772 26151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:01:50.899838 26151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:01:50.912243 26156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:01:50.912616 26159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:01:50.912678 26157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:01:50.913146 26151 server_base.cc:1034] running on GCE node
I20260924 16:01:50.913655 26151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:01:50.914287 26151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:01:50.915491 26151 hybrid_clock.cc:648] HybridClock initialized: now 1790265710915483 us; error 76 us; skew 500 ppm
Sep 24 16:01:50 dist-test-slave-n106 krb5kdc[26044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265710, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:01:50.918658 26151 init.cc:377] Logged in from keytab as kudu/127.25.109.1@KRBTEST.COM (short username kudu)
I20260924 16:01:50.919705 26151 webserver.cc:460] Webserver started at http://127.25.109.1:34733/ using document root <none> and password file <none>
I20260924 16:01:50.920289 26151 fs_manager.cc:359] Metadata directory not provided
I20260924 16:01:50.920441 26151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:01:50.920686 26151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:01:50.922850 26151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-0/data/instance:
uuid: "c61c3d49cf2f4e0fb051456e484df1c2"
format_stamp: "Formatted at 2026-09-24 16:01:50 on dist-test-slave-n106"
server_key: "bea516c47da9c341ac53942e6b59bb31"
server_key_iv: "360718ceebf243e208fa6d4595e09901"
server_key_version: "encryptionkey@0"
I20260924 16:01:50.923457 26151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance:
uuid: "c61c3d49cf2f4e0fb051456e484df1c2"
format_stamp: "Formatted at 2026-09-24 16:01:50 on dist-test-slave-n106"
server_key: "bea516c47da9c341ac53942e6b59bb31"
server_key_iv: "360718ceebf243e208fa6d4595e09901"
server_key_version: "encryptionkey@0"
I20260924 16:01:50.928025 26151 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260924 16:01:50.930982 26166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:50.932314 26151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:01:50.932482 26151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "c61c3d49cf2f4e0fb051456e484df1c2"
format_stamp: "Formatted at 2026-09-24 16:01:50 on dist-test-slave-n106"
server_key: "bea516c47da9c341ac53942e6b59bb31"
server_key_iv: "360718ceebf243e208fa6d4595e09901"
server_key_version: "encryptionkey@0"
I20260924 16:01:50.932628 26151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:01:50.945530 26151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:01:50.949123 26151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:01:50.949324 26151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:01:50.949903 26151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:01:50.950891 26151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:01:50.950985 26151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:50.951054 26151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:01:50.951100 26151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:50.962168 26151 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:44491
I20260924 16:01:50.962242 26281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:44491 every 8 connection(s)
I20260924 16:01:50.963505 26151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:01:50.965642 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 26151
I20260924 16:01:50.965744 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance
I20260924 16:01:50.966007 26036 external_mini_cluster.cc:1428] Setting key 948f3cee5783e96b8679be044173911b
WARNING: no policy specified for kudu/127.25.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.2@KRBTEST.COM" created.
Sep 24 16:01:50 dist-test-slave-n106 krb5kdc[26044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265710, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:01:50.984704 26128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:50.965609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:35539 (local address 127.25.109.62:38529)
0924 16:01:50.965748 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:50.965752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:50.966795 (+ 1043us) server_negotiation.cc:407] Connection header received
0924 16:01:50.970314 (+ 3519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:50.970317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:50.970376 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:50.970463 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:50.971932 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:50.972972 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:50.974361 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:50.974725 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:50.974846 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:50.980322 (+ 5476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:50.980336 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:50.980341 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:50.980398 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:50.982122 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:50.982805 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:50.982809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:50.982810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:50.982899 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:50.983348 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:50.983353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:50.983355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:50.983606 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:50.983687 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:50.984376 (+ 689us) server_negotiation.cc:299] Negotiation successful
0924 16:01:50.984531 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 16:01:50.985648 26284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:50.966044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:38529 (local address 127.25.109.1:35539)
0924 16:01:50.966627 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:50.966691 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:50.970061 (+ 3370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:50.970680 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:50.970687 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:50.971067 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:01:50.971733 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:50.971753 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:50.973119 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:50.973122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:50.974217 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:50.974229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:50.974900 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:50.974904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:50.975104 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:50.975811 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:01:50.975828 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:01:50.980076 (+ 4248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:01:50.982250 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:50.982260 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:50.982280 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:50.982682 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:50.982999 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:50.983005 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:50.983010 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:50.983222 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:50.983726 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:01:50.983733 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:01:50.984109 (+ 376us) client_negotiation.cc:769] Sending connection context
0924 16:01:50.984330 (+ 221us) client_negotiation.cc:240] Negotiation successful
0924 16:01:50.984716 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":144,"threads_started":1}
I20260924 16:01:50.986999 26282 heartbeater.cc:347] Connected to a master server at 127.25.109.62:38529
I20260924 16:01:50.987303 26282 heartbeater.cc:464] Registering TS with master...
I20260924 16:01:50.987840 26282 heartbeater.cc:511] Master 127.25.109.62:38529 requested a full tablet report, sending...
I20260924 16:01:50.989707 26089 ts_manager.cc:194] Registered new tserver with Master: c61c3d49cf2f4e0fb051456e484df1c2 (127.25.109.1:44491)
I20260924 16:01:50.990885 26089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.1@KRBTEST.COM'} at 127.25.109.1:35539
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
I20260924 16:01:50.995644 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:0
--local_ip_for_outbound_sockets=127.25.109.2
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:38529
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:35787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:01:51.134243 26287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:01:51.134505 26287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:01:51.134550 26287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:01:51.138753 26287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:01:51.138823 26287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:01:51.138883 26287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:01:51.144210 26287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:35787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:38529
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:01:51.145274 26287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:01:51.146293 26287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:01:51.157040 26292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:01:51.157126 26293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:01:51.157212 26287 server_base.cc:1034] running on GCE node
W20260924 16:01:51.157040 26295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:01:51.157809 26287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:01:51.158464 26287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:01:51.159667 26287 hybrid_clock.cc:648] HybridClock initialized: now 1790265711159631 us; error 48 us; skew 500 ppm
Sep 24 16:01:51 dist-test-slave-n106 krb5kdc[26044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265711, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:01:51.162501 26287 init.cc:377] Logged in from keytab as kudu/127.25.109.2@KRBTEST.COM (short username kudu)
I20260924 16:01:51.163378 26287 webserver.cc:460] Webserver started at http://127.25.109.2:39263/ using document root <none> and password file <none>
I20260924 16:01:51.163978 26287 fs_manager.cc:359] Metadata directory not provided
I20260924 16:01:51.164070 26287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:01:51.164305 26287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:01:51.166426 26287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-1/data/instance:
uuid: "3e340fcda35f4f9ea85d82a1131511ee"
format_stamp: "Formatted at 2026-09-24 16:01:51 on dist-test-slave-n106"
server_key: "d2ab173d4b4150c9fa84f1491f210d3c"
server_key_iv: "a21489dd48718e55b6b90405d4016e1a"
server_key_version: "encryptionkey@0"
I20260924 16:01:51.167009 26287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance:
uuid: "3e340fcda35f4f9ea85d82a1131511ee"
format_stamp: "Formatted at 2026-09-24 16:01:51 on dist-test-slave-n106"
server_key: "d2ab173d4b4150c9fa84f1491f210d3c"
server_key_iv: "a21489dd48718e55b6b90405d4016e1a"
server_key_version: "encryptionkey@0"
I20260924 16:01:51.171058 26287 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:01:51.174156 26302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:51.175441 26287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 16:01:51.175592 26287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "3e340fcda35f4f9ea85d82a1131511ee"
format_stamp: "Formatted at 2026-09-24 16:01:51 on dist-test-slave-n106"
server_key: "d2ab173d4b4150c9fa84f1491f210d3c"
server_key_iv: "a21489dd48718e55b6b90405d4016e1a"
server_key_version: "encryptionkey@0"
I20260924 16:01:51.175752 26287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:01:51.192286 26287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:01:51.195484 26287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:01:51.195679 26287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:01:51.196255 26287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:01:51.197206 26287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:01:51.197255 26287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:51.197335 26287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:01:51.197376 26287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:51.213488 26287 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:34489
I20260924 16:01:51.213506 26417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:34489 every 8 connection(s)
I20260924 16:01:51.214754 26287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:01:51.223165 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 26287
I20260924 16:01:51.223285 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance
I20260924 16:01:51.223605 26036 external_mini_cluster.cc:1428] Setting key f8813d17616b7ae3d0aedb63350b2716
Sep 24 16:01:51 dist-test-slave-n106 krb5kdc[26044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265711, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:01:51.230939 26128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:51.216928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:58565 (local address 127.25.109.62:38529)
0924 16:01:51.217149 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:51.217154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:51.218464 (+ 1310us) server_negotiation.cc:407] Connection header received
0924 16:01:51.219683 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:51.219686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:51.219733 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:51.219814 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:51.221264 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:51.221813 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:51.223101 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:51.223378 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:51.223509 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:51.226337 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:51.226341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:51.226343 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:51.226372 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:51.228359 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:51.228984 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:51.228988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:51.228990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:51.229100 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:51.229570 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:51.229573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:51.229574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:51.229764 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:51.229844 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:51.230688 (+ 844us) server_negotiation.cc:299] Negotiation successful
0924 16:01:51.230817 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260924 16:01:51.232358 26420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:51.217425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:38529 (local address 127.25.109.2:58565)
0924 16:01:51.218317 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:51.218360 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:51.219477 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:51.219992 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:51.219999 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:51.220519 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:01:51.221125 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:51.221140 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:51.221930 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:51.221933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:51.222959 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:51.222973 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:51.223576 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:51.223581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:51.223852 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:51.224598 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:01:51.224615 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:01:51.226166 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:01:51.228551 (+ 2385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:51.228557 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:51.228577 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:51.228874 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:51.229223 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:51.229230 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:51.229236 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:51.229451 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:51.229888 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:01:51.229900 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:01:51.230307 (+ 407us) client_negotiation.cc:769] Sending connection context
0924 16:01:51.230596 (+ 289us) client_negotiation.cc:240] Negotiation successful
0924 16:01:51.231037 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":123,"threads_started":1}
I20260924 16:01:51.233707 26418 heartbeater.cc:347] Connected to a master server at 127.25.109.62:38529
I20260924 16:01:51.234038 26418 heartbeater.cc:464] Registering TS with master...
I20260924 16:01:51.234802 26418 heartbeater.cc:511] Master 127.25.109.62:38529 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.25.109.3@KRBTEST.COM; defaulting to no policy
I20260924 16:01:51.236162 26089 ts_manager.cc:194] Registered new tserver with Master: 3e340fcda35f4f9ea85d82a1131511ee (127.25.109.2:34489)
I20260924 16:01:51.236935 26089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.2@KRBTEST.COM'} at 127.25.109.2:58565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
I20260924 16:01:51.253610 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:0
--local_ip_for_outbound_sockets=127.25.109.3
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:38529
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:35787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:01:51.386420 26423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:01:51.386703 26423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:01:51.386782 26423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:01:51.390923 26423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:01:51.391016 26423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:01:51.391111 26423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:01:51.396744 26423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:35787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:38529
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:01:51.397835 26423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:01:51.398957 26423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:01:51.408756 26428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:01:51.408845 26431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:01:51.408818 26423 server_base.cc:1034] running on GCE node
W20260924 16:01:51.408762 26429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:01:51.409297 26423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:01:51.409921 26423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:01:51.411116 26423 hybrid_clock.cc:648] HybridClock initialized: now 1790265711411075 us; error 47 us; skew 500 ppm
Sep 24 16:01:51 dist-test-slave-n106 krb5kdc[26044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265711, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:01:51.413796 26423 init.cc:377] Logged in from keytab as kudu/127.25.109.3@KRBTEST.COM (short username kudu)
I20260924 16:01:51.414572 26423 webserver.cc:460] Webserver started at http://127.25.109.3:42197/ using document root <none> and password file <none>
I20260924 16:01:51.415086 26423 fs_manager.cc:359] Metadata directory not provided
I20260924 16:01:51.415146 26423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:01:51.415309 26423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:01:51.417565 26423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-2/data/instance:
uuid: "26cc878cecd14808bce57d1b6a018d6e"
format_stamp: "Formatted at 2026-09-24 16:01:51 on dist-test-slave-n106"
server_key: "645a3e781d9f799a93b5db2ba3bf1d4f"
server_key_iv: "8b9f556661195f9f0557c3efa36d7925"
server_key_version: "encryptionkey@0"
I20260924 16:01:51.418200 26423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance:
uuid: "26cc878cecd14808bce57d1b6a018d6e"
format_stamp: "Formatted at 2026-09-24 16:01:51 on dist-test-slave-n106"
server_key: "645a3e781d9f799a93b5db2ba3bf1d4f"
server_key_iv: "8b9f556661195f9f0557c3efa36d7925"
server_key_version: "encryptionkey@0"
I20260924 16:01:51.422401 26423 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:01:51.424945 26438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:51.426216 26423 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:01:51.426312 26423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "26cc878cecd14808bce57d1b6a018d6e"
format_stamp: "Formatted at 2026-09-24 16:01:51 on dist-test-slave-n106"
server_key: "645a3e781d9f799a93b5db2ba3bf1d4f"
server_key_iv: "8b9f556661195f9f0557c3efa36d7925"
server_key_version: "encryptionkey@0"
I20260924 16:01:51.426400 26423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:01:51.445309 26423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:01:51.448370 26423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:01:51.448604 26423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:01:51.449208 26423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:01:51.450181 26423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:01:51.450229 26423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:51.450301 26423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:01:51.450342 26423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:51.463110 26423 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:37475
I20260924 16:01:51.463231 26552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:37475 every 8 connection(s)
I20260924 16:01:51.464201 26423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:01:51.470088 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 26423
I20260924 16:01:51.470254 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance
I20260924 16:01:51.470602 26036 external_mini_cluster.cc:1428] Setting key 4e70145237b553b0b99ff10189953765
Sep 24 16:01:51 dist-test-slave-n106 krb5kdc[26044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265711, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:01:51.480201 26128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:51.466603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:58217 (local address 127.25.109.62:38529)
0924 16:01:51.466750 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:51.466753 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:51.467983 (+ 1230us) server_negotiation.cc:407] Connection header received
0924 16:01:51.469385 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:51.469389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:51.469453 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:51.469593 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:51.471659 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:51.472327 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:51.473697 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:51.473919 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:51.474034 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:51.476788 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:51.476792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:51.476794 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:51.476819 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:51.478101 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:51.478613 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:51.478618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:51.478620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:51.478701 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:51.479055 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:51.479059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:51.479060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:51.479324 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:51.479432 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:51.479922 (+ 490us) server_negotiation.cc:299] Negotiation successful
0924 16:01:51.480042 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 16:01:51.481158 26555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:51.466966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:38529 (local address 127.25.109.3:58217)
0924 16:01:51.467824 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:51.467868 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:51.469044 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:51.469805 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:51.469816 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:51.470570 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:01:51.471356 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:51.471378 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:51.472523 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:51.472526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:51.473566 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:51.473577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:51.474073 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:51.474078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:51.474307 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:51.474945 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:01:51.474974 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:01:51.476615 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:01:51.478220 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:51.478226 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:51.478242 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:51.478503 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:51.478794 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:51.478798 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:51.478800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:51.478941 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:51.479429 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:01:51.479437 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:01:51.479678 (+ 241us) client_negotiation.cc:769] Sending connection context
0924 16:01:51.479869 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 16:01:51.480202 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":183,"threads_started":1}
I20260924 16:01:51.482327 26553 heartbeater.cc:347] Connected to a master server at 127.25.109.62:38529
I20260924 16:01:51.482587 26553 heartbeater.cc:464] Registering TS with master...
I20260924 16:01:51.483079 26553 heartbeater.cc:511] Master 127.25.109.62:38529 requested a full tablet report, sending...
I20260924 16:01:51.484211 26089 ts_manager.cc:194] Registered new tserver with Master: 26cc878cecd14808bce57d1b6a018d6e (127.25.109.3:37475)
I20260924 16:01:51.484929 26089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.3@KRBTEST.COM'} at 127.25.109.3:58217
I20260924 16:01:51.485337 26036 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 16:01:51 dist-test-slave-n106 krb5kdc[26044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265711, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 16:01:51 dist-test-slave-n106 krb5kdc[26044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265711, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.109.1@KRBTEST.COM
I20260924 16:01:51.511286 26560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:51.501888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52018 (local address 127.25.109.1:44491)
0924 16:01:51.502207 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:51.502213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:51.502239 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:01:51.502330 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:51.502335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:51.502537 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:51.502665 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:51.503285 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:51.504067 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:51.504941 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:51.505313 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:51.505401 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:51.507809 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:51.507818 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:51.507832 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:51.507856 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:51.509610 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:51.509995 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:51.509999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:51.510001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:51.510091 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:51.510322 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:51.510325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:51.510326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:51.510568 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:51.510736 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:51.510911 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 16:01:51.511085 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":78,"threads_started":1}
W20260924 16:01:51.512347 26194 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52018
Sep 24 16:01:51 dist-test-slave-n106 krb5kdc[26044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 16:01:51 dist-test-slave-n106 krb5kdc[26044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.1@KRBTEST.COM
I20260924 16:01:51.538827 26560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:51.530177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52022 (local address 127.25.109.1:44491)
0924 16:01:51.530349 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:51.530356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:51.530787 (+ 431us) server_negotiation.cc:407] Connection header received
0924 16:01:51.530979 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:51.530984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:51.531053 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:51.531165 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:51.532001 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:51.532609 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:51.533434 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:51.533663 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:51.533737 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:51.536165 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:51.536169 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:51.536171 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:51.536194 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:51.537465 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:51.537873 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:51.537876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:51.537877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:51.537949 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:51.538173 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:51.538176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:51.538177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:51.538322 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:51.538431 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:51.538614 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 16:01:51.538704 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 16:01:51.539911 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 26151
I20260924 16:01:51.547209 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 26287
I20260924 16:01:51.553913 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 26423
I20260924 16:01:51.559077 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 26057
2026-09-24T16:01:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2439 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:01:51 dist-test-slave-n106 krb5kdc[26567](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:01:51 dist-test-slave-n106 krb5kdc[26567](info): set up 2 sockets
Sep 24 16:01:51 dist-test-slave-n106 krb5kdc[26567](info): commencing operation
krb5kdc: starting...
W20260924 16:01:52.600958 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:01:52 dist-test-slave-n106 krb5kdc[26567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:01:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:01:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
I20260924 16:01:52.723270 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:34883
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:39477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:34883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:01:52.834347 26580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:01:52.834623 26580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:01:52.834705 26580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:01:52.838805 26580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:01:52.838865 26580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:01:52.838886 26580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:01:52.838902 26580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:01:52.838946 26580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:01:52.844218 26580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:39477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:34883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:34883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:01:52.845311 26580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:01:52.846382 26580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:01:52.855072 26588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:01:52.855093 26585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:01:52.855217 26580 server_base.cc:1034] running on GCE node
W20260924 16:01:52.855135 26586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:01:52.855962 26580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:01:52.857085 26580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:01:52.858245 26580 hybrid_clock.cc:648] HybridClock initialized: now 1790265712858218 us; error 39 us; skew 500 ppm
Sep 24 16:01:52 dist-test-slave-n106 krb5kdc[26567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265712, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:01:52.866834 26580 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:01:52.867674 26580 webserver.cc:460] Webserver started at http://127.25.109.62:41895/ using document root <none> and password file <none>
I20260924 16:01:52.868219 26580 fs_manager.cc:359] Metadata directory not provided
I20260924 16:01:52.868278 26580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:01:52.868526 26580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:01:52.870484 26580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "af98ef2aa94d40bfa6dedbbe272c1f94"
format_stamp: "Formatted at 2026-09-24 16:01:52 on dist-test-slave-n106"
server_key: "eae74b81fde7477741729c25dfa79deb"
server_key_iv: "df29ff6c63de8ec8b7c1436cfe40e06c"
server_key_version: "encryptionkey@0"
I20260924 16:01:52.870997 26580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "af98ef2aa94d40bfa6dedbbe272c1f94"
format_stamp: "Formatted at 2026-09-24 16:01:52 on dist-test-slave-n106"
server_key: "eae74b81fde7477741729c25dfa79deb"
server_key_iv: "df29ff6c63de8ec8b7c1436cfe40e06c"
server_key_version: "encryptionkey@0"
I20260924 16:01:52.874616 26580 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 16:01:52.876926 26595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:52.878129 26580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:01:52.878222 26580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "af98ef2aa94d40bfa6dedbbe272c1f94"
format_stamp: "Formatted at 2026-09-24 16:01:52 on dist-test-slave-n106"
server_key: "eae74b81fde7477741729c25dfa79deb"
server_key_iv: "df29ff6c63de8ec8b7c1436cfe40e06c"
server_key_version: "encryptionkey@0"
I20260924 16:01:52.878322 26580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:01:52.889927 26580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:01:52.893184 26580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:01:52.893333 26580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:01:52.900432 26580 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:34883
I20260924 16:01:52.900498 26647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:34883 every 8 connection(s)
I20260924 16:01:52.901422 26580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:01:52.904198 26648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:01:52.909662 26648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af98ef2aa94d40bfa6dedbbe272c1f94: Bootstrap starting.
I20260924 16:01:52.909847 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 26580
I20260924 16:01:52.909977 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:01:52.910218 26036 external_mini_cluster.cc:1428] Setting key c0cd61abd7cd6d5d6b58b60ff58db7c1
I20260924 16:01:52.912252 26648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af98ef2aa94d40bfa6dedbbe272c1f94: Neither blocks nor log segments found. Creating new log.
I20260924 16:01:52.913057 26648 log.cc:824] T 00000000000000000000000000000000 P af98ef2aa94d40bfa6dedbbe272c1f94: Log is configured to *not* fsync() on all Append() calls
I20260924 16:01:52.915143 26648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af98ef2aa94d40bfa6dedbbe272c1f94: No bootstrap required, opened a new log
Sep 24 16:01:52 dist-test-slave-n106 krb5kdc[26567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:01:52.919641 26648 raft_consensus.cc:348] T 00000000000000000000000000000000 P af98ef2aa94d40bfa6dedbbe272c1f94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af98ef2aa94d40bfa6dedbbe272c1f94" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 34883 } }
I20260924 16:01:52.919888 26648 raft_consensus.cc:374] T 00000000000000000000000000000000 P af98ef2aa94d40bfa6dedbbe272c1f94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:01:52.919972 26648 raft_consensus.cc:729] T 00000000000000000000000000000000 P af98ef2aa94d40bfa6dedbbe272c1f94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af98ef2aa94d40bfa6dedbbe272c1f94, State: Initialized, Role: FOLLOWER
I20260924 16:01:52.920537 26648 consensus_queue.cc:260] T 00000000000000000000000000000000 P af98ef2aa94d40bfa6dedbbe272c1f94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af98ef2aa94d40bfa6dedbbe272c1f94" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 34883 } }
I20260924 16:01:52.920725 26648 raft_consensus.cc:388] T 00000000000000000000000000000000 P af98ef2aa94d40bfa6dedbbe272c1f94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:01:52.920823 26648 raft_consensus.cc:482] T 00000000000000000000000000000000 P af98ef2aa94d40bfa6dedbbe272c1f94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:01:52.920934 26648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af98ef2aa94d40bfa6dedbbe272c1f94 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:01:52.921808 26648 raft_consensus.cc:504] T 00000000000000000000000000000000 P af98ef2aa94d40bfa6dedbbe272c1f94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af98ef2aa94d40bfa6dedbbe272c1f94" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 34883 } }
I20260924 16:01:52.922163 26648 leader_election.cc:302] T 00000000000000000000000000000000 P af98ef2aa94d40bfa6dedbbe272c1f94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af98ef2aa94d40bfa6dedbbe272c1f94; no voters:
I20260924 16:01:52.922505 26648 leader_election.cc:288] T 00000000000000000000000000000000 P af98ef2aa94d40bfa6dedbbe272c1f94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:01:52.922569 26653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af98ef2aa94d40bfa6dedbbe272c1f94 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:01:52.922801 26653 raft_consensus.cc:686] T 00000000000000000000000000000000 P af98ef2aa94d40bfa6dedbbe272c1f94 [term 1 LEADER]: Becoming Leader. State: Replica: af98ef2aa94d40bfa6dedbbe272c1f94, State: Running, Role: LEADER
I20260924 16:01:52.923100 26653 consensus_queue.cc:237] T 00000000000000000000000000000000 P af98ef2aa94d40bfa6dedbbe272c1f94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af98ef2aa94d40bfa6dedbbe272c1f94" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 34883 } }
I20260924 16:01:52.923866 26648 sys_catalog.cc:564] T 00000000000000000000000000000000 P af98ef2aa94d40bfa6dedbbe272c1f94 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:01:52.924274 26653 sys_catalog.cc:455] T 00000000000000000000000000000000 P af98ef2aa94d40bfa6dedbbe272c1f94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af98ef2aa94d40bfa6dedbbe272c1f94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af98ef2aa94d40bfa6dedbbe272c1f94" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 34883 } } }
I20260924 16:01:52.924391 26653 sys_catalog.cc:458] T 00000000000000000000000000000000 P af98ef2aa94d40bfa6dedbbe272c1f94 [sys.catalog]: This master's current role is: LEADER
I20260924 16:01:52.924842 26657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:01:52.924865 26654 sys_catalog.cc:455] T 00000000000000000000000000000000 P af98ef2aa94d40bfa6dedbbe272c1f94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af98ef2aa94d40bfa6dedbbe272c1f94. Latest consensus state: current_term: 1 leader_uuid: "af98ef2aa94d40bfa6dedbbe272c1f94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af98ef2aa94d40bfa6dedbbe272c1f94" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 34883 } } }
I20260924 16:01:52.925056 26654 sys_catalog.cc:458] T 00000000000000000000000000000000 P af98ef2aa94d40bfa6dedbbe272c1f94 [sys.catalog]: This master's current role is: LEADER
I20260924 16:01:52.925264 26651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:52.911553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50538 (local address 127.25.109.62:34883)
0924 16:01:52.911954 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:52.911970 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:52.912003 (+ 33us) server_negotiation.cc:407] Connection header received
0924 16:01:52.912715 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:52.912735 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:52.913068 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:52.913390 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:52.914230 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:52.915187 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:52.916351 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:52.916871 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:52.916993 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:52.919768 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:52.919773 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:52.919787 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:52.919812 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:52.921695 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:52.922194 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:52.922200 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:52.922206 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:52.922363 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:52.922754 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:52.922758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:52.922759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:52.923301 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:52.923527 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:52.923835 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 16:01:52.924169 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":130,"threads_started":1}
I20260924 16:01:52.929113 26657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:01:52.934762 26657 catalog_manager.cc:1347] Generated new cluster ID: 64b7b8dfd7254688848185dffef9d3c8
I20260924 16:01:52.934827 26657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:01:52.949615 26657 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:01:52.950534 26657 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:01:52.956835 26657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af98ef2aa94d40bfa6dedbbe272c1f94: Generated new TSK 0
I20260924 16:01:52.957589 26657 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
I20260924 16:01:52.991801 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:0
--local_ip_for_outbound_sockets=127.25.109.1
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:34883
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:39477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:01:53.101958 26674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:01:53.102217 26674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:01:53.102260 26674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:01:53.106269 26674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:01:53.106323 26674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:01:53.106422 26674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:01:53.111519 26674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:39477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:34883
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:01:53.112596 26674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:01:53.113651 26674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:01:53.123399 26681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:01:53.123469 26680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:01:53.123461 26674 server_base.cc:1034] running on GCE node
W20260924 16:01:53.123265 26683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:01:53.124044 26674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:01:53.124822 26674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:01:53.126015 26674 hybrid_clock.cc:648] HybridClock initialized: now 1790265713125998 us; error 27 us; skew 500 ppm
Sep 24 16:01:53 dist-test-slave-n106 krb5kdc[26567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265713, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:01:53.128870 26674 init.cc:377] Logged in from keytab as kudu/127.25.109.1@KRBTEST.COM (short username kudu)
I20260924 16:01:53.129731 26674 webserver.cc:460] Webserver started at http://127.25.109.1:39249/ using document root <none> and password file <none>
I20260924 16:01:53.130316 26674 fs_manager.cc:359] Metadata directory not provided
I20260924 16:01:53.130419 26674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:01:53.130671 26674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:01:53.132920 26674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-0/data/instance:
uuid: "4448dc8a191847788bebbbeccfeb7654"
format_stamp: "Formatted at 2026-09-24 16:01:53 on dist-test-slave-n106"
server_key: "05645d5fb32d0db3855ff1bfe50f156e"
server_key_iv: "a368b9bd33a844f61bdcf534342a9a76"
server_key_version: "encryptionkey@0"
I20260924 16:01:53.133530 26674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance:
uuid: "4448dc8a191847788bebbbeccfeb7654"
format_stamp: "Formatted at 2026-09-24 16:01:53 on dist-test-slave-n106"
server_key: "05645d5fb32d0db3855ff1bfe50f156e"
server_key_iv: "a368b9bd33a844f61bdcf534342a9a76"
server_key_version: "encryptionkey@0"
I20260924 16:01:53.137380 26674 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:01:53.139935 26690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:53.141275 26674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 16:01:53.141407 26674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "4448dc8a191847788bebbbeccfeb7654"
format_stamp: "Formatted at 2026-09-24 16:01:53 on dist-test-slave-n106"
server_key: "05645d5fb32d0db3855ff1bfe50f156e"
server_key_iv: "a368b9bd33a844f61bdcf534342a9a76"
server_key_version: "encryptionkey@0"
I20260924 16:01:53.141552 26674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:01:53.168028 26674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:01:53.171352 26674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:01:53.171582 26674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:01:53.172231 26674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:01:53.173162 26674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:01:53.173219 26674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:53.173296 26674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:01:53.173328 26674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:53.182600 26674 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:34501
I20260924 16:01:53.182623 26803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:34501 every 8 connection(s)
I20260924 16:01:53.183570 26674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:01:53.188279 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 26674
I20260924 16:01:53.188370 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance
I20260924 16:01:53.188663 26036 external_mini_cluster.cc:1428] Setting key 2f4e777599072799af75db95cf253f44
Sep 24 16:01:53 dist-test-slave-n106 krb5kdc[26567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265713, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:01:53.198560 26651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:53.185541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:42657 (local address 127.25.109.62:34883)
0924 16:01:53.185688 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:53.185692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:53.186524 (+ 832us) server_negotiation.cc:407] Connection header received
0924 16:01:53.187877 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:53.187881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:53.187945 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:53.188059 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:53.189508 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.190140 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.191288 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.191524 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.191648 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:53.194354 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:53.194361 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:53.194367 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:53.194401 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:53.196254 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:53.196864 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:53.196869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:53.196870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:53.196973 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:53.197339 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:53.197343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:53.197345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:53.197599 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:53.197711 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:53.198213 (+ 502us) server_negotiation.cc:299] Negotiation successful
0924 16:01:53.198350 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.25.109.2@KRBTEST.COM; defaulting to no policy
I20260924 16:01:53.199359 26806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:53.185893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:34883 (local address 127.25.109.1:42657)
0924 16:01:53.186367 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:53.186410 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:53.187652 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:53.188268 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:53.188276 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:53.188773 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:01:53.189341 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:53.189361 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.190266 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.190269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:53.191156 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:53.191165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.191687 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.191691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:53.191863 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:53.192513 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:01:53.192529 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:01:53.194176 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:01:53.196392 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:53.196398 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:53.196472 (+ 74us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:53.196753 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:53.197078 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:53.197080 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:53.197081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:53.197223 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:53.197707 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:01:53.197712 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:01:53.197945 (+ 233us) client_negotiation.cc:769] Sending connection context
0924 16:01:53.198107 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 16:01:53.198367 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":121,"threads_started":1}
I20260924 16:01:53.200803 26804 heartbeater.cc:347] Connected to a master server at 127.25.109.62:34883
I20260924 16:01:53.201095 26804 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.2@KRBTEST.COM" created.
I20260924 16:01:53.201695 26804 heartbeater.cc:511] Master 127.25.109.62:34883 requested a full tablet report, sending...
I20260924 16:01:53.203470 26612 ts_manager.cc:194] Registered new tserver with Master: 4448dc8a191847788bebbbeccfeb7654 (127.25.109.1:34501)
I20260924 16:01:53.204689 26612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.1@KRBTEST.COM'} at 127.25.109.1:42657
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
I20260924 16:01:53.219796 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:0
--local_ip_for_outbound_sockets=127.25.109.2
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:34883
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:39477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:01:53.332392 26809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:01:53.332657 26809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:01:53.332700 26809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:01:53.336802 26809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:01:53.336855 26809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:01:53.336913 26809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:01:53.341962 26809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:39477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:34883
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:01:53.342945 26809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:01:53.343920 26809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:01:53.353786 26817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:01:53.353775 26815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:01:53.353855 26809 server_base.cc:1034] running on GCE node
W20260924 16:01:53.353778 26814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:01:53.354391 26809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:01:53.355046 26809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:01:53.356242 26809 hybrid_clock.cc:648] HybridClock initialized: now 1790265713356225 us; error 28 us; skew 500 ppm
Sep 24 16:01:53 dist-test-slave-n106 krb5kdc[26567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265713, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:01:53.358964 26809 init.cc:377] Logged in from keytab as kudu/127.25.109.2@KRBTEST.COM (short username kudu)
I20260924 16:01:53.359853 26809 webserver.cc:460] Webserver started at http://127.25.109.2:40511/ using document root <none> and password file <none>
I20260924 16:01:53.360476 26809 fs_manager.cc:359] Metadata directory not provided
I20260924 16:01:53.360569 26809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:01:53.360797 26809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:01:53.362823 26809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-1/data/instance:
uuid: "deb17e3fba4943a7b7815420d3fde978"
format_stamp: "Formatted at 2026-09-24 16:01:53 on dist-test-slave-n106"
server_key: "a8b66b8f72c001d41749523d75dd8b44"
server_key_iv: "36283e092eb5c12fc16bc7d05c42aad4"
server_key_version: "encryptionkey@0"
I20260924 16:01:53.363396 26809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance:
uuid: "deb17e3fba4943a7b7815420d3fde978"
format_stamp: "Formatted at 2026-09-24 16:01:53 on dist-test-slave-n106"
server_key: "a8b66b8f72c001d41749523d75dd8b44"
server_key_iv: "36283e092eb5c12fc16bc7d05c42aad4"
server_key_version: "encryptionkey@0"
I20260924 16:01:53.367136 26809 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 16:01:53.369649 26825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:53.370849 26809 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:01:53.370986 26809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "deb17e3fba4943a7b7815420d3fde978"
format_stamp: "Formatted at 2026-09-24 16:01:53 on dist-test-slave-n106"
server_key: "a8b66b8f72c001d41749523d75dd8b44"
server_key_iv: "36283e092eb5c12fc16bc7d05c42aad4"
server_key_version: "encryptionkey@0"
I20260924 16:01:53.371110 26809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:01:53.382537 26809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:01:53.385316 26809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:01:53.385519 26809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:01:53.386097 26809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:01:53.386921 26809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:01:53.386997 26809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:53.387074 26809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:01:53.387130 26809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:53.395985 26809 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:43369
I20260924 16:01:53.396023 26938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:43369 every 8 connection(s)
I20260924 16:01:53.396926 26809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:01:53.406239 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 26809
I20260924 16:01:53.406381 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance
Sep 24 16:01:53 dist-test-slave-n106 krb5kdc[26567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265713, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:01:53.406765 26036 external_mini_cluster.cc:1428] Setting key 829c41a558ea2bfe3d6378175ff7a16e
I20260924 16:01:53.410770 26651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:53.398764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:34133 (local address 127.25.109.62:34883)
0924 16:01:53.398903 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:53.398906 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:53.399658 (+ 752us) server_negotiation.cc:407] Connection header received
0924 16:01:53.401081 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:53.401084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:53.401145 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:53.401268 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:53.402430 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.402966 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.404142 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.404373 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.404552 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:53.407086 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:53.407091 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:53.407094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:53.407128 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:53.408703 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:53.409219 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:53.409223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:53.409225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:53.409312 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:53.409666 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:53.409670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:53.409671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:53.409854 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:53.409968 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:53.410466 (+ 498us) server_negotiation.cc:299] Negotiation successful
0924 16:01:53.410591 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 16:01:53.411625 26941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:53.399070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:34883 (local address 127.25.109.2:34133)
0924 16:01:53.399511 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:53.399552 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:53.400857 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:53.401449 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:53.401456 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:53.401795 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:01:53.402266 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:53.402280 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.403113 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.403121 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:53.404000 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:53.404010 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.404596 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.404600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:53.404777 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:53.405391 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:01:53.405407 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:01:53.406927 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:01:53.408821 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:53.408827 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:53.408842 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:53.409109 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:53.409411 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:53.409415 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:53.409417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:53.409561 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:53.409967 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:01:53.409973 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:01:53.410223 (+ 250us) client_negotiation.cc:769] Sending connection context
0924 16:01:53.410417 (+ 194us) client_negotiation.cc:240] Negotiation successful
0924 16:01:53.410747 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":112,"threads_started":1}
I20260924 16:01:53.412848 26939 heartbeater.cc:347] Connected to a master server at 127.25.109.62:34883
I20260924 16:01:53.413111 26939 heartbeater.cc:464] Registering TS with master...
I20260924 16:01:53.413617 26939 heartbeater.cc:511] Master 127.25.109.62:34883 requested a full tablet report, sending...
I20260924 16:01:53.414776 26612 ts_manager.cc:194] Registered new tserver with Master: deb17e3fba4943a7b7815420d3fde978 (127.25.109.2:43369)
I20260924 16:01:53.415616 26612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.2@KRBTEST.COM'} at 127.25.109.2:34133
WARNING: no policy specified for kudu/127.25.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
I20260924 16:01:53.435926 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:0
--local_ip_for_outbound_sockets=127.25.109.3
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:34883
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:39477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:01:53.548142 26944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:01:53.548420 26944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:01:53.548466 26944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:01:53.552461 26944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:01:53.552521 26944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:01:53.552582 26944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:01:53.557783 26944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:39477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:34883
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:01:53.558811 26944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:01:53.559775 26944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:01:53.569505 26949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:01:53.569505 26950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:01:53.569504 26952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:01:53.569854 26944 server_base.cc:1034] running on GCE node
I20260924 16:01:53.570257 26944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:01:53.570900 26944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:01:53.572093 26944 hybrid_clock.cc:648] HybridClock initialized: now 1790265713572065 us; error 51 us; skew 500 ppm
Sep 24 16:01:53 dist-test-slave-n106 krb5kdc[26567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265713, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:01:53.574939 26944 init.cc:377] Logged in from keytab as kudu/127.25.109.3@KRBTEST.COM (short username kudu)
I20260924 16:01:53.575789 26944 webserver.cc:460] Webserver started at http://127.25.109.3:33421/ using document root <none> and password file <none>
I20260924 16:01:53.576372 26944 fs_manager.cc:359] Metadata directory not provided
I20260924 16:01:53.576500 26944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:01:53.576689 26944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:01:53.578660 26944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-2/data/instance:
uuid: "1dd5e34f525b4b9c9647a606590eb366"
format_stamp: "Formatted at 2026-09-24 16:01:53 on dist-test-slave-n106"
server_key: "fbb805ef7973be2e6812826dbadac3b1"
server_key_iv: "dd6c93020ea45a704bd575b8c3c7f7b3"
server_key_version: "encryptionkey@0"
I20260924 16:01:53.579167 26944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance:
uuid: "1dd5e34f525b4b9c9647a606590eb366"
format_stamp: "Formatted at 2026-09-24 16:01:53 on dist-test-slave-n106"
server_key: "fbb805ef7973be2e6812826dbadac3b1"
server_key_iv: "dd6c93020ea45a704bd575b8c3c7f7b3"
server_key_version: "encryptionkey@0"
I20260924 16:01:53.583145 26944 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 16:01:53.585584 26959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:53.586779 26944 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:01:53.586877 26944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "1dd5e34f525b4b9c9647a606590eb366"
format_stamp: "Formatted at 2026-09-24 16:01:53 on dist-test-slave-n106"
server_key: "fbb805ef7973be2e6812826dbadac3b1"
server_key_iv: "dd6c93020ea45a704bd575b8c3c7f7b3"
server_key_version: "encryptionkey@0"
I20260924 16:01:53.587013 26944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:01:53.599153 26944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:01:53.602172 26944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:01:53.602396 26944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:01:53.603024 26944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:01:53.603901 26944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:01:53.603947 26944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:53.603988 26944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:01:53.604053 26944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:53.613348 26944 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:40337
I20260924 16:01:53.613366 27074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:40337 every 8 connection(s)
I20260924 16:01:53.614279 26944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:01:53.622495 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 26944
I20260924 16:01:53.622620 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance
I20260924 16:01:53.622926 26036 external_mini_cluster.cc:1428] Setting key d1922fc5535994044238a84790f0e99b
Sep 24 16:01:53 dist-test-slave-n106 krb5kdc[26567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265713, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:01:53.627866 26651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:53.616110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:40803 (local address 127.25.109.62:34883)
0924 16:01:53.616252 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:53.616256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:53.617076 (+ 820us) server_negotiation.cc:407] Connection header received
0924 16:01:53.618043 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:53.618046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:53.618119 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:53.618228 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:53.619431 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.619975 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.621218 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.621449 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.621569 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:53.624440 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:53.624444 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:53.624448 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:53.624476 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:53.626061 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:53.626544 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:53.626546 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:53.626547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:53.626627 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:53.626931 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:53.626933 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:53.626935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:53.627113 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:53.627187 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:53.627652 (+ 465us) server_negotiation.cc:299] Negotiation successful
0924 16:01:53.627746 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 16:01:53.628820 27077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:53.616458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:34883 (local address 127.25.109.3:40803)
0924 16:01:53.616913 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:53.616956 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:53.617822 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:53.618422 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:53.618430 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:53.618799 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:01:53.619278 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:53.619291 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.620112 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.620116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:53.621084 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:53.621094 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.621626 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.621631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:53.621799 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:53.622441 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:01:53.622458 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:01:53.624250 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:01:53.626169 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:53.626174 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:53.626187 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:53.626443 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:53.626710 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:53.626713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:53.626714 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:53.626849 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:53.627203 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:01:53.627208 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:01:53.627438 (+ 230us) client_negotiation.cc:769] Sending connection context
0924 16:01:53.627600 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 16:01:53.627859 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":114,"threads_started":1}
I20260924 16:01:53.630003 27075 heartbeater.cc:347] Connected to a master server at 127.25.109.62:34883
I20260924 16:01:53.630232 27075 heartbeater.cc:464] Registering TS with master...
I20260924 16:01:53.630668 27075 heartbeater.cc:511] Master 127.25.109.62:34883 requested a full tablet report, sending...
I20260924 16:01:53.631673 26612 ts_manager.cc:194] Registered new tserver with Master: 1dd5e34f525b4b9c9647a606590eb366 (127.25.109.3:40337)
I20260924 16:01:53.632256 26612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.3@KRBTEST.COM'} at 127.25.109.3:40803
I20260924 16:01:53.637079 26036 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:01:53.649664 26651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:53.642268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50546 (local address 127.25.109.62:34883)
0924 16:01:53.642425 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:53.642429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:53.642499 (+ 70us) server_negotiation.cc:407] Connection header received
0924 16:01:53.642607 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:53.642609 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:53.642651 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:53.642715 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:53.643390 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.643923 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.644794 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.645003 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.645084 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:53.646529 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:53.646533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:53.646535 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:53.646558 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:53.647880 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:53.648310 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:53.648314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:53.648316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:53.648399 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:53.648750 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:53.648754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:53.648756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:53.648983 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:53.649125 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:53.649368 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 16:01:53.649484 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 16:01:53.655934 26612 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:01:53.659162 26612 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:01:53.674382 27086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:53.668326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:34501 (local address 127.0.0.1:56472)
0924 16:01:53.668713 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:53.668772 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:53.668900 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:53.670171 (+ 1271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:53.670177 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:53.670200 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:01:53.670315 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:53.670319 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.671518 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.671525 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:53.672577 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:53.672587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.673835 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.673838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:53.673938 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:53.673977 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 16:01:53.674101 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 16:01:53.674228 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":96,"threads_started":1}
I20260924 16:01:53.674399 27089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:53.669087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56472 (local address 127.25.109.1:34501)
0924 16:01:53.669412 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:53.669444 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:53.669471 (+ 27us) server_negotiation.cc:407] Connection header received
0924 16:01:53.669554 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:53.669559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:53.669787 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:53.669949 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:01:53.670468 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.671328 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.672711 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.673686 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.673812 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:53.673970 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:53.674133 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 16:01:53.674268 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":92,"threads_started":1}
I20260924 16:01:53.675315 27088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:53.668431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:40337 (local address 127.0.0.1:44166)
0924 16:01:53.669571 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:53.669585 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:53.669675 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:53.670399 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:53.670405 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:53.670426 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:01:53.670564 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:53.670572 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.672305 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.672308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:53.673551 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:53.673560 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.674929 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.674932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:53.675021 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:53.675053 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 16:01:53.675113 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 16:01:53.675182 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1059,"mutex_wait_us":32,"thread_start_us":94,"threads_started":1}
I20260924 16:01:53.675407 27090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:53.669221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44166 (local address 127.25.109.3:40337)
0924 16:01:53.669935 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:53.669946 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:53.669969 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:01:53.670046 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:53.670053 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:53.670249 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:53.670406 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:01:53.670986 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.672125 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.673679 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.674806 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.674922 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:53.675081 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:53.675150 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 16:01:53.675242 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":90,"threads_started":1}
I20260924 16:01:53.675791 27087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:53.668696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:43369 (local address 127.0.0.1:41888)
0924 16:01:53.669854 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:53.669866 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:53.669944 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:53.670704 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:53.670707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:53.670726 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:01:53.670818 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:53.670825 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.672437 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.672440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:53.673680 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:53.673689 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.675089 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.675095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:53.675205 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:53.675229 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 16:01:53.675277 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 16:01:53.675348 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1098,"mutex_wait_us":77,"thread_start_us":87,"threads_started":1}
I20260924 16:01:53.676240 27091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:53.669203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41888 (local address 127.25.109.2:43369)
0924 16:01:53.670261 (+ 1058us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:53.670274 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:53.670296 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:01:53.670372 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:53.670376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:53.670541 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:53.670713 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:01:53.670946 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.672123 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.673805 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.674960 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.675771 (+ 811us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:53.675932 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:53.675983 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 16:01:53.676086 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":884,"thread_start_us":243,"threads_started":1}
I20260924 16:01:53.678426 27008 tablet_service.cc:1467] Processing CreateTablet for tablet 7a0a29e4409c4098a985694c0d650469 (DEFAULT_TABLE table=test-table [id=9c2f55b5edbb4d4d92cd310e208b88aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:01:53.678499 26738 tablet_service.cc:1467] Processing CreateTablet for tablet 7a0a29e4409c4098a985694c0d650469 (DEFAULT_TABLE table=test-table [id=9c2f55b5edbb4d4d92cd310e208b88aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:01:53.678815 26873 tablet_service.cc:1467] Processing CreateTablet for tablet 7a0a29e4409c4098a985694c0d650469 (DEFAULT_TABLE table=test-table [id=9c2f55b5edbb4d4d92cd310e208b88aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:01:53.679718 27008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a0a29e4409c4098a985694c0d650469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:01:53.679718 26738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a0a29e4409c4098a985694c0d650469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:01:53.679756 26873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a0a29e4409c4098a985694c0d650469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:01:53.686853 27092 tablet_bootstrap.cc:492] T 7a0a29e4409c4098a985694c0d650469 P deb17e3fba4943a7b7815420d3fde978: Bootstrap starting.
I20260924 16:01:53.687903 27093 tablet_bootstrap.cc:492] T 7a0a29e4409c4098a985694c0d650469 P 4448dc8a191847788bebbbeccfeb7654: Bootstrap starting.
I20260924 16:01:53.688469 27094 tablet_bootstrap.cc:492] T 7a0a29e4409c4098a985694c0d650469 P 1dd5e34f525b4b9c9647a606590eb366: Bootstrap starting.
I20260924 16:01:53.690093 27093 tablet_bootstrap.cc:654] T 7a0a29e4409c4098a985694c0d650469 P 4448dc8a191847788bebbbeccfeb7654: Neither blocks nor log segments found. Creating new log.
I20260924 16:01:53.692703 27094 tablet_bootstrap.cc:654] T 7a0a29e4409c4098a985694c0d650469 P 1dd5e34f525b4b9c9647a606590eb366: Neither blocks nor log segments found. Creating new log.
I20260924 16:01:53.692703 27092 tablet_bootstrap.cc:654] T 7a0a29e4409c4098a985694c0d650469 P deb17e3fba4943a7b7815420d3fde978: Neither blocks nor log segments found. Creating new log.
I20260924 16:01:53.693403 27093 log.cc:824] T 7a0a29e4409c4098a985694c0d650469 P 4448dc8a191847788bebbbeccfeb7654: Log is configured to *not* fsync() on all Append() calls
I20260924 16:01:53.693859 27094 log.cc:824] T 7a0a29e4409c4098a985694c0d650469 P 1dd5e34f525b4b9c9647a606590eb366: Log is configured to *not* fsync() on all Append() calls
I20260924 16:01:53.693923 27092 log.cc:824] T 7a0a29e4409c4098a985694c0d650469 P deb17e3fba4943a7b7815420d3fde978: Log is configured to *not* fsync() on all Append() calls
I20260924 16:01:53.695562 27093 tablet_bootstrap.cc:492] T 7a0a29e4409c4098a985694c0d650469 P 4448dc8a191847788bebbbeccfeb7654: No bootstrap required, opened a new log
I20260924 16:01:53.695787 27093 ts_tablet_manager.cc:1397] T 7a0a29e4409c4098a985694c0d650469 P 4448dc8a191847788bebbbeccfeb7654: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260924 16:01:53.695902 27092 tablet_bootstrap.cc:492] T 7a0a29e4409c4098a985694c0d650469 P deb17e3fba4943a7b7815420d3fde978: No bootstrap required, opened a new log
I20260924 16:01:53.696099 27092 ts_tablet_manager.cc:1397] T 7a0a29e4409c4098a985694c0d650469 P deb17e3fba4943a7b7815420d3fde978: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260924 16:01:53.696219 27094 tablet_bootstrap.cc:492] T 7a0a29e4409c4098a985694c0d650469 P 1dd5e34f525b4b9c9647a606590eb366: No bootstrap required, opened a new log
I20260924 16:01:53.696398 27094 ts_tablet_manager.cc:1397] T 7a0a29e4409c4098a985694c0d650469 P 1dd5e34f525b4b9c9647a606590eb366: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260924 16:01:53.699064 27093 raft_consensus.cc:348] T 7a0a29e4409c4098a985694c0d650469 P 4448dc8a191847788bebbbeccfeb7654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4448dc8a191847788bebbbeccfeb7654" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 34501 } } peers { permanent_uuid: "1dd5e34f525b4b9c9647a606590eb366" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40337 } } peers { permanent_uuid: "deb17e3fba4943a7b7815420d3fde978" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43369 } }
I20260924 16:01:53.699375 27093 raft_consensus.cc:374] T 7a0a29e4409c4098a985694c0d650469 P 4448dc8a191847788bebbbeccfeb7654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:01:53.699431 27093 raft_consensus.cc:729] T 7a0a29e4409c4098a985694c0d650469 P 4448dc8a191847788bebbbeccfeb7654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4448dc8a191847788bebbbeccfeb7654, State: Initialized, Role: FOLLOWER
I20260924 16:01:53.699963 27093 consensus_queue.cc:260] T 7a0a29e4409c4098a985694c0d650469 P 4448dc8a191847788bebbbeccfeb7654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4448dc8a191847788bebbbeccfeb7654" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 34501 } } peers { permanent_uuid: "1dd5e34f525b4b9c9647a606590eb366" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40337 } } peers { permanent_uuid: "deb17e3fba4943a7b7815420d3fde978" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43369 } }
I20260924 16:01:53.700626 26804 heartbeater.cc:503] Master 127.25.109.62:34883 was elected leader, sending a full tablet report...
I20260924 16:01:53.700999 27093 ts_tablet_manager.cc:1428] T 7a0a29e4409c4098a985694c0d650469 P 4448dc8a191847788bebbbeccfeb7654: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 16:01:53.701237 27094 raft_consensus.cc:348] T 7a0a29e4409c4098a985694c0d650469 P 1dd5e34f525b4b9c9647a606590eb366 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4448dc8a191847788bebbbeccfeb7654" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 34501 } } peers { permanent_uuid: "1dd5e34f525b4b9c9647a606590eb366" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40337 } } peers { permanent_uuid: "deb17e3fba4943a7b7815420d3fde978" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43369 } }
I20260924 16:01:53.701237 27092 raft_consensus.cc:348] T 7a0a29e4409c4098a985694c0d650469 P deb17e3fba4943a7b7815420d3fde978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4448dc8a191847788bebbbeccfeb7654" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 34501 } } peers { permanent_uuid: "1dd5e34f525b4b9c9647a606590eb366" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40337 } } peers { permanent_uuid: "deb17e3fba4943a7b7815420d3fde978" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43369 } }
I20260924 16:01:53.701488 27094 raft_consensus.cc:374] T 7a0a29e4409c4098a985694c0d650469 P 1dd5e34f525b4b9c9647a606590eb366 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:01:53.701498 27092 raft_consensus.cc:374] T 7a0a29e4409c4098a985694c0d650469 P deb17e3fba4943a7b7815420d3fde978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:01:53.701551 27092 raft_consensus.cc:729] T 7a0a29e4409c4098a985694c0d650469 P deb17e3fba4943a7b7815420d3fde978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deb17e3fba4943a7b7815420d3fde978, State: Initialized, Role: FOLLOWER
I20260924 16:01:53.701560 27094 raft_consensus.cc:729] T 7a0a29e4409c4098a985694c0d650469 P 1dd5e34f525b4b9c9647a606590eb366 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dd5e34f525b4b9c9647a606590eb366, State: Initialized, Role: FOLLOWER
I20260924 16:01:53.701978 27094 consensus_queue.cc:260] T 7a0a29e4409c4098a985694c0d650469 P 1dd5e34f525b4b9c9647a606590eb366 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4448dc8a191847788bebbbeccfeb7654" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 34501 } } peers { permanent_uuid: "1dd5e34f525b4b9c9647a606590eb366" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40337 } } peers { permanent_uuid: "deb17e3fba4943a7b7815420d3fde978" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43369 } }
I20260924 16:01:53.701978 27092 consensus_queue.cc:260] T 7a0a29e4409c4098a985694c0d650469 P deb17e3fba4943a7b7815420d3fde978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4448dc8a191847788bebbbeccfeb7654" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 34501 } } peers { permanent_uuid: "1dd5e34f525b4b9c9647a606590eb366" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40337 } } peers { permanent_uuid: "deb17e3fba4943a7b7815420d3fde978" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43369 } }
I20260924 16:01:53.702847 27075 heartbeater.cc:503] Master 127.25.109.62:34883 was elected leader, sending a full tablet report...
I20260924 16:01:53.702883 26939 heartbeater.cc:503] Master 127.25.109.62:34883 was elected leader, sending a full tablet report...
I20260924 16:01:53.702927 27094 ts_tablet_manager.cc:1428] T 7a0a29e4409c4098a985694c0d650469 P 1dd5e34f525b4b9c9647a606590eb366: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 16:01:53.703867 27092 ts_tablet_manager.cc:1428] T 7a0a29e4409c4098a985694c0d650469 P deb17e3fba4943a7b7815420d3fde978: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260924 16:01:53.706214 27100 raft_consensus.cc:482] T 7a0a29e4409c4098a985694c0d650469 P 1dd5e34f525b4b9c9647a606590eb366 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:01:53.706429 27100 raft_consensus.cc:504] T 7a0a29e4409c4098a985694c0d650469 P 1dd5e34f525b4b9c9647a606590eb366 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4448dc8a191847788bebbbeccfeb7654" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 34501 } } peers { permanent_uuid: "1dd5e34f525b4b9c9647a606590eb366" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40337 } } peers { permanent_uuid: "deb17e3fba4943a7b7815420d3fde978" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43369 } }
I20260924 16:01:53.707615 27100 leader_election.cc:288] T 7a0a29e4409c4098a985694c0d650469 P 1dd5e34f525b4b9c9647a606590eb366 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4448dc8a191847788bebbbeccfeb7654 (127.25.109.1:34501), deb17e3fba4943a7b7815420d3fde978 (127.25.109.2:43369)
I20260924 16:01:53.711440 27077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:53.707794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:34501 (local address 127.25.109.3:56559)
0924 16:01:53.707901 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:53.707918 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:53.708008 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:53.708597 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:53.708600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:53.708624 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:01:53.708735 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:53.708742 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.709466 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.709469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:53.710333 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:53.710342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.711004 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.711009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:53.711116 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:53.711139 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 16:01:53.711213 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 16:01:53.711299 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":51}
I20260924 16:01:53.711463 27089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:53.708175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:56559 (local address 127.25.109.1:34501)
0924 16:01:53.708324 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:53.708328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:53.708340 (+ 12us) server_negotiation.cc:407] Connection header received
0924 16:01:53.708407 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:53.708412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:53.708490 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:53.708566 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:01:53.708851 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.709316 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.710470 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.710875 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.710970 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:53.711048 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:53.711246 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 16:01:53.711332 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:01:53.711509 27091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:53.707932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:53377 (local address 127.25.109.2:43369)
0924 16:01:53.708159 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:53.708163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:53.708177 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:01:53.708305 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:53.708308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:53.708344 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:53.708451 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:01:53.708731 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.709280 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.710494 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.710906 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.711045 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:53.711098 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:53.711331 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 16:01:53.711401 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260924 16:01:53.711565 27102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:53.707859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:43369 (local address 127.25.109.3:53377)
0924 16:01:53.708107 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:53.708118 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:53.708191 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:53.708459 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:53.708462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:53.708479 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:01:53.708616 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:53.708625 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.709390 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.709397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:53.710340 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:53.710349 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.711057 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.711063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:53.711173 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:53.711218 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 16:01:53.711298 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 16:01:53.711357 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":62,"thread_start_us":107,"threads_started":1}
I20260924 16:01:53.713025 26893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a0a29e4409c4098a985694c0d650469" candidate_uuid: "1dd5e34f525b4b9c9647a606590eb366" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deb17e3fba4943a7b7815420d3fde978" is_pre_election: true
I20260924 16:01:53.713050 26758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a0a29e4409c4098a985694c0d650469" candidate_uuid: "1dd5e34f525b4b9c9647a606590eb366" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4448dc8a191847788bebbbeccfeb7654" is_pre_election: true
I20260924 16:01:53.713332 26893 raft_consensus.cc:2454] T 7a0a29e4409c4098a985694c0d650469 P deb17e3fba4943a7b7815420d3fde978 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dd5e34f525b4b9c9647a606590eb366 in term 0.
I20260924 16:01:53.713332 26758 raft_consensus.cc:2454] T 7a0a29e4409c4098a985694c0d650469 P 4448dc8a191847788bebbbeccfeb7654 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dd5e34f525b4b9c9647a606590eb366 in term 0.
I20260924 16:01:53.713922 26961 leader_election.cc:302] T 7a0a29e4409c4098a985694c0d650469 P 1dd5e34f525b4b9c9647a606590eb366 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1dd5e34f525b4b9c9647a606590eb366, 4448dc8a191847788bebbbeccfeb7654, deb17e3fba4943a7b7815420d3fde978; no voters:
I20260924 16:01:53.714226 27100 raft_consensus.cc:2781] T 7a0a29e4409c4098a985694c0d650469 P 1dd5e34f525b4b9c9647a606590eb366 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:01:53.714305 27100 raft_consensus.cc:482] T 7a0a29e4409c4098a985694c0d650469 P 1dd5e34f525b4b9c9647a606590eb366 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:01:53.714336 27100 raft_consensus.cc:3037] T 7a0a29e4409c4098a985694c0d650469 P 1dd5e34f525b4b9c9647a606590eb366 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:01:53.715423 27100 raft_consensus.cc:504] T 7a0a29e4409c4098a985694c0d650469 P 1dd5e34f525b4b9c9647a606590eb366 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4448dc8a191847788bebbbeccfeb7654" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 34501 } } peers { permanent_uuid: "1dd5e34f525b4b9c9647a606590eb366" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40337 } } peers { permanent_uuid: "deb17e3fba4943a7b7815420d3fde978" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43369 } }
I20260924 16:01:53.715771 27100 leader_election.cc:288] T 7a0a29e4409c4098a985694c0d650469 P 1dd5e34f525b4b9c9647a606590eb366 [CANDIDATE]: Term 1 election: Requested vote from peers 4448dc8a191847788bebbbeccfeb7654 (127.25.109.1:34501), deb17e3fba4943a7b7815420d3fde978 (127.25.109.2:43369)
I20260924 16:01:53.716172 26893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a0a29e4409c4098a985694c0d650469" candidate_uuid: "1dd5e34f525b4b9c9647a606590eb366" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deb17e3fba4943a7b7815420d3fde978"
I20260924 16:01:53.716272 26758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a0a29e4409c4098a985694c0d650469" candidate_uuid: "1dd5e34f525b4b9c9647a606590eb366" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4448dc8a191847788bebbbeccfeb7654"
I20260924 16:01:53.716316 26893 raft_consensus.cc:3037] T 7a0a29e4409c4098a985694c0d650469 P deb17e3fba4943a7b7815420d3fde978 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:01:53.716394 26758 raft_consensus.cc:3037] T 7a0a29e4409c4098a985694c0d650469 P 4448dc8a191847788bebbbeccfeb7654 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:01:53.718151 26758 raft_consensus.cc:2454] T 7a0a29e4409c4098a985694c0d650469 P 4448dc8a191847788bebbbeccfeb7654 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dd5e34f525b4b9c9647a606590eb366 in term 1.
I20260924 16:01:53.718305 26893 raft_consensus.cc:2454] T 7a0a29e4409c4098a985694c0d650469 P deb17e3fba4943a7b7815420d3fde978 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dd5e34f525b4b9c9647a606590eb366 in term 1.
I20260924 16:01:53.718534 26963 leader_election.cc:302] T 7a0a29e4409c4098a985694c0d650469 P 1dd5e34f525b4b9c9647a606590eb366 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dd5e34f525b4b9c9647a606590eb366, 4448dc8a191847788bebbbeccfeb7654; no voters:
I20260924 16:01:53.718691 27100 raft_consensus.cc:2781] T 7a0a29e4409c4098a985694c0d650469 P 1dd5e34f525b4b9c9647a606590eb366 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:01:53.718876 27100 raft_consensus.cc:686] T 7a0a29e4409c4098a985694c0d650469 P 1dd5e34f525b4b9c9647a606590eb366 [term 1 LEADER]: Becoming Leader. State: Replica: 1dd5e34f525b4b9c9647a606590eb366, State: Running, Role: LEADER
I20260924 16:01:53.719210 27100 consensus_queue.cc:237] T 7a0a29e4409c4098a985694c0d650469 P 1dd5e34f525b4b9c9647a606590eb366 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4448dc8a191847788bebbbeccfeb7654" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 34501 } } peers { permanent_uuid: "1dd5e34f525b4b9c9647a606590eb366" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40337 } } peers { permanent_uuid: "deb17e3fba4943a7b7815420d3fde978" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43369 } }
I20260924 16:01:53.722568 26610 catalog_manager.cc:5495] T 7a0a29e4409c4098a985694c0d650469 P 1dd5e34f525b4b9c9647a606590eb366 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dd5e34f525b4b9c9647a606590eb366 (127.25.109.3). New cstate: current_term: 1 leader_uuid: "1dd5e34f525b4b9c9647a606590eb366" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4448dc8a191847788bebbbeccfeb7654" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 34501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dd5e34f525b4b9c9647a606590eb366" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "deb17e3fba4943a7b7815420d3fde978" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43369 } health_report { overall_health: UNKNOWN } } }
Sep 24 16:01:53 dist-test-slave-n106 krb5kdc[26567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.1@KRBTEST.COM
I20260924 16:01:53.741258 27089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:53.731953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56476 (local address 127.25.109.1:34501)
0924 16:01:53.732117 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:53.732123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:53.732335 (+ 212us) server_negotiation.cc:407] Connection header received
0924 16:01:53.732530 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:53.732536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:53.732590 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:53.732682 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:53.733347 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.733963 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.734859 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.735091 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.735211 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:53.737760 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:53.737772 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:53.737786 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:53.737810 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:53.739506 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:53.739938 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:53.739943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:53.739944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:53.740066 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:53.740392 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:53.740394 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:53.740395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:53.740685 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:53.740842 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:53.741028 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 16:01:53.741138 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 16:01:53 dist-test-slave-n106 krb5kdc[26567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265713, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 16:01:53 dist-test-slave-n106 krb5kdc[26567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265713, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.109.1@KRBTEST.COM
I20260924 16:01:53.769107 27089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:53.760229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56490 (local address 127.25.109.1:34501)
0924 16:01:53.760454 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:53.760459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:53.760567 (+ 108us) server_negotiation.cc:407] Connection header received
0924 16:01:53.760765 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:53.760768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:53.760822 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:53.760916 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:53.761629 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.762274 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.763148 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.763448 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.763569 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:53.766204 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:53.766208 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:53.766210 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:53.766235 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:53.767462 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:53.767839 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:53.767842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:53.767843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:53.767915 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:53.768205 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:53.768207 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:53.768209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:53.768371 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:53.768621 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:53.768875 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 16:01:53.768977 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
Sep 24 16:01:53 dist-test-slave-n106 krb5kdc[26567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 16:01:53 dist-test-slave-n106 krb5kdc[26567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.1@KRBTEST.COM
I20260924 16:01:53.795107 27089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:53.786678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56500 (local address 127.25.109.1:34501)
0924 16:01:53.786862 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:53.786867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:53.786997 (+ 130us) server_negotiation.cc:407] Connection header received
0924 16:01:53.787170 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:53.787174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:53.787226 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:53.787321 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:53.787991 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.788588 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.789437 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:53.789724 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:53.789854 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:53.792155 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:53.792159 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:53.792162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:53.792195 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:53.793440 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:53.793860 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:53.793865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:53.793867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:53.793948 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:53.794251 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:53.794255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:53.794257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:53.794421 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:53.794563 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:53.794806 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 16:01:53.794928 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260924 16:01:53.796718 26718 tablet_service.cc:2908] Rejecting scan request for tablet 7a0a29e4409c4098a985694c0d650469: Uninitialized: clean time has not yet been initialized
I20260924 16:01:53.798812 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 26674
I20260924 16:01:53.805712 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 26809
I20260924 16:01:53.812052 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 26944
I20260924 16:01:53.818111 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 26580
2026-09-24T16:01:53Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2259 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:01:53 dist-test-slave-n106 krb5kdc[27117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:01:53 dist-test-slave-n106 krb5kdc[27117](info): set up 2 sockets
Sep 24 16:01:53 dist-test-slave-n106 krb5kdc[27117](info): commencing operation
krb5kdc: starting...
W20260924 16:01:54.881211 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:01:54 dist-test-slave-n106 krb5kdc[27117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:01:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:01:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
I20260924 16:01:55.003727 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:33761
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:43535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:33761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:01:55.112778 27130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:01:55.113072 27130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:01:55.113158 27130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:01:55.117660 27130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:01:55.117753 27130 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 16:01:55.117784 27130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:01:55.117825 27130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:01:55.117864 27130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:01:55.117916 27130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:01:55.123176 27130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:43535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:33761
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:33761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:01:55.124243 27130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:01:55.125310 27130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:01:55.133831 27136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:01:55.133940 27130 server_base.cc:1034] running on GCE node
W20260924 16:01:55.133883 27135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:01:55.133854 27138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:01:55.134739 27130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:01:55.135958 27130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:01:55.137192 27130 hybrid_clock.cc:648] HybridClock initialized: now 1790265715137158 us; error 46 us; skew 500 ppm
Sep 24 16:01:55 dist-test-slave-n106 krb5kdc[27117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265715, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:01:55.140053 27130 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:01:55.141048 27130 webserver.cc:460] Webserver started at http://127.25.109.62:37263/ using document root <none> and password file <none>
I20260924 16:01:55.141721 27130 fs_manager.cc:359] Metadata directory not provided
I20260924 16:01:55.141839 27130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:01:55.142133 27130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:01:55.144330 27130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "973b347f0b28409f991e362af9865bfe"
format_stamp: "Formatted at 2026-09-24 16:01:55 on dist-test-slave-n106"
server_key: "f5a69537b4b1ad224d02c01a299056ab"
server_key_iv: "9dddfd7416bc6501926c48b726eb129c"
server_key_version: "encryptionkey@0"
I20260924 16:01:55.145020 27130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "973b347f0b28409f991e362af9865bfe"
format_stamp: "Formatted at 2026-09-24 16:01:55 on dist-test-slave-n106"
server_key: "f5a69537b4b1ad224d02c01a299056ab"
server_key_iv: "9dddfd7416bc6501926c48b726eb129c"
server_key_version: "encryptionkey@0"
I20260924 16:01:55.148866 27130 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:01:55.151494 27145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:55.152793 27130 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:01:55.152948 27130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "973b347f0b28409f991e362af9865bfe"
format_stamp: "Formatted at 2026-09-24 16:01:55 on dist-test-slave-n106"
server_key: "f5a69537b4b1ad224d02c01a299056ab"
server_key_iv: "9dddfd7416bc6501926c48b726eb129c"
server_key_version: "encryptionkey@0"
I20260924 16:01:55.153079 27130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:01:55.171627 27130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:01:55.174744 27130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:01:55.174975 27130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:01:55.183807 27130 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:33761
I20260924 16:01:55.183837 27208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:33761 every 8 connection(s)
I20260924 16:01:55.184937 27130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:01:55.187713 27209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:01:55.189479 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 27130
I20260924 16:01:55.189579 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:01:55.189862 26036 external_mini_cluster.cc:1428] Setting key df8cbf1d9e9b87086728ea3003ba7c81
I20260924 16:01:55.194581 27209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 973b347f0b28409f991e362af9865bfe: Bootstrap starting.
I20260924 16:01:55.197221 27209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 973b347f0b28409f991e362af9865bfe: Neither blocks nor log segments found. Creating new log.
I20260924 16:01:55.198033 27209 log.cc:824] T 00000000000000000000000000000000 P 973b347f0b28409f991e362af9865bfe: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:01:55 dist-test-slave-n106 krb5kdc[27117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:01:55.200201 27209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 973b347f0b28409f991e362af9865bfe: No bootstrap required, opened a new log
I20260924 16:01:55.203140 27209 raft_consensus.cc:348] T 00000000000000000000000000000000 P 973b347f0b28409f991e362af9865bfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "973b347f0b28409f991e362af9865bfe" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33761 } }
I20260924 16:01:55.203402 27209 raft_consensus.cc:374] T 00000000000000000000000000000000 P 973b347f0b28409f991e362af9865bfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:01:55.203505 27209 raft_consensus.cc:729] T 00000000000000000000000000000000 P 973b347f0b28409f991e362af9865bfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 973b347f0b28409f991e362af9865bfe, State: Initialized, Role: FOLLOWER
I20260924 16:01:55.204018 27209 consensus_queue.cc:260] T 00000000000000000000000000000000 P 973b347f0b28409f991e362af9865bfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "973b347f0b28409f991e362af9865bfe" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33761 } }
I20260924 16:01:55.204192 27209 raft_consensus.cc:388] T 00000000000000000000000000000000 P 973b347f0b28409f991e362af9865bfe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:01:55.204282 27209 raft_consensus.cc:482] T 00000000000000000000000000000000 P 973b347f0b28409f991e362af9865bfe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:01:55.204384 27209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 973b347f0b28409f991e362af9865bfe [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:01:55.205469 27209 raft_consensus.cc:504] T 00000000000000000000000000000000 P 973b347f0b28409f991e362af9865bfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "973b347f0b28409f991e362af9865bfe" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33761 } }
I20260924 16:01:55.205813 27209 leader_election.cc:302] T 00000000000000000000000000000000 P 973b347f0b28409f991e362af9865bfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 973b347f0b28409f991e362af9865bfe; no voters:
I20260924 16:01:55.206135 27209 leader_election.cc:288] T 00000000000000000000000000000000 P 973b347f0b28409f991e362af9865bfe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:01:55.206195 27214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 973b347f0b28409f991e362af9865bfe [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:01:55.206410 27214 raft_consensus.cc:686] T 00000000000000000000000000000000 P 973b347f0b28409f991e362af9865bfe [term 1 LEADER]: Becoming Leader. State: Replica: 973b347f0b28409f991e362af9865bfe, State: Running, Role: LEADER
I20260924 16:01:55.206431 27212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:55.191308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48704 (local address 127.25.109.62:33761)
0924 16:01:55.191755 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:55.191773 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:55.191817 (+ 44us) server_negotiation.cc:407] Connection header received
0924 16:01:55.192699 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:55.192724 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:55.193142 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:55.193513 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:55.194273 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.195717 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.196743 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.197449 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.197623 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:55.200462 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:55.200473 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:55.200493 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:55.200524 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:55.203007 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:55.203596 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:55.203605 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:55.203614 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:55.203744 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:55.204085 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:55.204088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:55.204089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:55.204653 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:55.204816 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:55.205131 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 16:01:55.205460 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":121,"threads_started":1}
I20260924 16:01:55.206780 27214 consensus_queue.cc:237] T 00000000000000000000000000000000 P 973b347f0b28409f991e362af9865bfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "973b347f0b28409f991e362af9865bfe" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33761 } }
I20260924 16:01:55.207420 27209 sys_catalog.cc:564] T 00000000000000000000000000000000 P 973b347f0b28409f991e362af9865bfe [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:01:55.209327 27216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 973b347f0b28409f991e362af9865bfe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 973b347f0b28409f991e362af9865bfe. Latest consensus state: current_term: 1 leader_uuid: "973b347f0b28409f991e362af9865bfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "973b347f0b28409f991e362af9865bfe" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33761 } } }
I20260924 16:01:55.209463 27216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 973b347f0b28409f991e362af9865bfe [sys.catalog]: This master's current role is: LEADER
I20260924 16:01:55.209862 27215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 973b347f0b28409f991e362af9865bfe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "973b347f0b28409f991e362af9865bfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "973b347f0b28409f991e362af9865bfe" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33761 } } }
I20260924 16:01:55.210220 27215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 973b347f0b28409f991e362af9865bfe [sys.catalog]: This master's current role is: LEADER
W20260924 16:01:55.212308 27228 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 973b347f0b28409f991e362af9865bfe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:01:55.212374 27228 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:01:55.212492 27229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:01:55.213567 27229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:01:55.219172 27229 catalog_manager.cc:1347] Generated new cluster ID: 3e53e8825a044a10a3cce5ec2cfee172
I20260924 16:01:55.219237 27229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:01:55.226959 27229 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:01:55.228140 27229 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:01:55.239454 27229 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 973b347f0b28409f991e362af9865bfe: Generated new TSK 0
I20260924 16:01:55.240254 27229 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
I20260924 16:01:55.275044 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:0
--local_ip_for_outbound_sockets=127.25.109.1
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:33761
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:43535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:01:55.389183 27235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:01:55.389426 27235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:01:55.389508 27235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:01:55.393533 27235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:01:55.393587 27235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:01:55.393658 27235 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:01:55.393743 27235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:01:55.398939 27235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:43535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.109.62:33761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:01:55.399971 27235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:01:55.401017 27235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:01:55.410367 27243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:01:55.410384 27240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:01:55.410382 27241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:01:55.410723 27235 server_base.cc:1034] running on GCE node
I20260924 16:01:55.411049 27235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:01:55.411636 27235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:01:55.412798 27235 hybrid_clock.cc:648] HybridClock initialized: now 1790265715412783 us; error 23 us; skew 500 ppm
Sep 24 16:01:55 dist-test-slave-n106 krb5kdc[27117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265715, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:01:55.415349 27235 init.cc:377] Logged in from keytab as kudu/127.25.109.1@KRBTEST.COM (short username kudu)
I20260924 16:01:55.416175 27235 webserver.cc:460] Webserver started at http://127.25.109.1:44341/ using document root <none> and password file <none>
I20260924 16:01:55.416793 27235 fs_manager.cc:359] Metadata directory not provided
I20260924 16:01:55.416889 27235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:01:55.417122 27235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:01:55.419102 27235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-0/data/instance:
uuid: "3df6aff1795d498bbfbd1046d78be724"
format_stamp: "Formatted at 2026-09-24 16:01:55 on dist-test-slave-n106"
server_key: "c80c0389bdae542a6899597292b4be29"
server_key_iv: "906cbbca8d62e7ee044de6a497fc6f49"
server_key_version: "encryptionkey@0"
I20260924 16:01:55.419682 27235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance:
uuid: "3df6aff1795d498bbfbd1046d78be724"
format_stamp: "Formatted at 2026-09-24 16:01:55 on dist-test-slave-n106"
server_key: "c80c0389bdae542a6899597292b4be29"
server_key_iv: "906cbbca8d62e7ee044de6a497fc6f49"
server_key_version: "encryptionkey@0"
I20260924 16:01:55.425794 27235 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 16:01:55.429035 27250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:55.430815 27235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:01:55.430956 27235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "3df6aff1795d498bbfbd1046d78be724"
format_stamp: "Formatted at 2026-09-24 16:01:55 on dist-test-slave-n106"
server_key: "c80c0389bdae542a6899597292b4be29"
server_key_iv: "906cbbca8d62e7ee044de6a497fc6f49"
server_key_version: "encryptionkey@0"
I20260924 16:01:55.431146 27235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:01:55.448252 27235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:01:55.451059 27235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:01:55.451284 27235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:01:55.452958 27235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:01:55.453043 27235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:55.453109 27235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:01:55.453163 27235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:01:55 dist-test-slave-n106 krb5kdc[27117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265715, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:01:55.463625 27235 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:40269
I20260924 16:01:55.463660 27366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:40269 every 8 connection(s)
I20260924 16:01:55.464635 27235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:01:55.468266 27212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:55.453822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:58179 (local address 127.25.109.62:33761)
0924 16:01:55.454000 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:55.454007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:55.454852 (+ 845us) server_negotiation.cc:407] Connection header received
0924 16:01:55.456051 (+ 1199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:55.456057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:55.456131 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:55.456242 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:55.457844 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.458733 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.460611 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.460866 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.460991 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:55.464111 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:55.464116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:55.464119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:55.464156 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:55.466164 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:55.466704 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:55.466708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:55.466710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:55.466800 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:55.467164 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:55.467168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:55.467169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:55.467358 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:55.467471 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:55.467954 (+ 483us) server_negotiation.cc:299] Negotiation successful
0924 16:01:55.468082 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 16:01:55.469043 27259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:55.454211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33761 (local address 127.25.109.1:58179)
0924 16:01:55.454682 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:55.454734 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:55.455788 (+ 1054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:55.456506 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:55.456516 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:55.456985 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:01:55.457692 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:55.457715 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.458926 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.458932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:55.460490 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:55.460506 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.461034 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.461038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:55.461223 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:55.461953 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:01:55.461978 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:01:55.463925 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:01:55.466293 (+ 2368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:55.466299 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:55.466314 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:55.466572 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:55.466904 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:55.466908 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:55.466910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:55.467052 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:55.467471 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:01:55.467479 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:01:55.467720 (+ 241us) client_negotiation.cc:769] Sending connection context
0924 16:01:55.467910 (+ 190us) client_negotiation.cc:240] Negotiation successful
0924 16:01:55.468223 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":135,"threads_started":1}
I20260924 16:01:55.470294 27367 heartbeater.cc:347] Connected to a master server at 127.25.109.62:33761
I20260924 16:01:55.470546 27367 heartbeater.cc:464] Registering TS with master...
I20260924 16:01:55.471042 27367 heartbeater.cc:511] Master 127.25.109.62:33761 requested a full tablet report, sending...
I20260924 16:01:55.472362 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 27235
I20260924 16:01:55.472555 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance
I20260924 16:01:55.472824 27162 ts_manager.cc:194] Registered new tserver with Master: 3df6aff1795d498bbfbd1046d78be724 (127.25.109.1:40269)
I20260924 16:01:55.472899 26036 external_mini_cluster.cc:1428] Setting key e22629a397847e0042b37358b89e9403
I20260924 16:01:55.474606 27162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.1@KRBTEST.COM'} at 127.25.109.1:58179
I20260924 16:01:55.482214 27373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:55.472685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33761 (local address 127.25.109.1:48675)
0924 16:01:55.472979 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:55.472994 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:55.473081 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:55.473321 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:55.473324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:55.473529 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:01:55.473648 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:55.473654 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.474528 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.474534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:55.477282 (+ 2748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:55.477291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.477894 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.477897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:55.477972 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:55.478468 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:01:55.478479 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:01:55.478785 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:01:55.480791 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:55.480793 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:55.480795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:55.480987 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:55.481286 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:55.481289 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:55.481290 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:55.481365 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:55.481804 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:01:55.481806 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:01:55.481879 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 16:01:55.481958 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 16:01:55.482060 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":101,"threads_started":1}
I20260924 16:01:55.482362 27212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:55.473016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:48675 (local address 127.25.109.62:33761)
0924 16:01:55.473121 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:55.473125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:55.473138 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:01:55.473173 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:55.473176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:55.473217 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:55.473303 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:55.473764 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.474401 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.477408 (+ 3007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.477776 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.477897 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:55.478938 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:55.478945 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:55.478947 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:55.478977 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:55.480679 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:55.481089 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:55.481094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:55.481096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:55.481188 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:55.481460 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:55.481464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:55.481468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:55.481701 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:55.481813 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:55.482057 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 16:01:55.482187 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":42}
WARNING: no policy specified for kudu/127.25.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
I20260924 16:01:55.501521 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:0
--local_ip_for_outbound_sockets=127.25.109.2
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:33761
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:43535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:01:55.613384 27376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:01:55.613679 27376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:01:55.613755 27376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:01:55.617784 27376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:01:55.617870 27376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:01:55.617942 27376 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:01:55.618031 27376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:01:55.623256 27376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:43535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.109.62:33761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:01:55.624357 27376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:01:55.625413 27376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:01:55.634918 27382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:01:55.634924 27384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:01:55.634999 27381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:01:55.635197 27376 server_base.cc:1034] running on GCE node
I20260924 16:01:55.635686 27376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:01:55.636370 27376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:01:55.637574 27376 hybrid_clock.cc:648] HybridClock initialized: now 1790265715637561 us; error 25 us; skew 500 ppm
Sep 24 16:01:55 dist-test-slave-n106 krb5kdc[27117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265715, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:01:55.640523 27376 init.cc:377] Logged in from keytab as kudu/127.25.109.2@KRBTEST.COM (short username kudu)
I20260924 16:01:55.641345 27376 webserver.cc:460] Webserver started at http://127.25.109.2:36231/ using document root <none> and password file <none>
I20260924 16:01:55.641858 27376 fs_manager.cc:359] Metadata directory not provided
I20260924 16:01:55.641913 27376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:01:55.642073 27376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:01:55.644034 27376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-1/data/instance:
uuid: "134b3b306d91453ebf1ed1314b21d3a8"
format_stamp: "Formatted at 2026-09-24 16:01:55 on dist-test-slave-n106"
server_key: "26b99928635f9a77997705de7c0acd0a"
server_key_iv: "72ab6b969dde997ec21709ea1986908e"
server_key_version: "encryptionkey@0"
I20260924 16:01:55.644618 27376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance:
uuid: "134b3b306d91453ebf1ed1314b21d3a8"
format_stamp: "Formatted at 2026-09-24 16:01:55 on dist-test-slave-n106"
server_key: "26b99928635f9a77997705de7c0acd0a"
server_key_iv: "72ab6b969dde997ec21709ea1986908e"
server_key_version: "encryptionkey@0"
I20260924 16:01:55.648052 27376 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:01:55.650405 27391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:55.651618 27376 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:01:55.651738 27376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "134b3b306d91453ebf1ed1314b21d3a8"
format_stamp: "Formatted at 2026-09-24 16:01:55 on dist-test-slave-n106"
server_key: "26b99928635f9a77997705de7c0acd0a"
server_key_iv: "72ab6b969dde997ec21709ea1986908e"
server_key_version: "encryptionkey@0"
I20260924 16:01:55.651844 27376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:01:55.664536 27376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:01:55.667227 27376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:01:55.667423 27376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:01:55.669003 27376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:01:55.669052 27376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:55.669116 27376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:01:55.669148 27376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:01:55 dist-test-slave-n106 krb5kdc[27117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265715, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:01:55.679888 27376 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:43127
I20260924 16:01:55.679942 27509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:43127 every 8 connection(s)
I20260924 16:01:55.681046 27376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:01:55.682968 27212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:55.669729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:46291 (local address 127.25.109.62:33761)
0924 16:01:55.669893 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:55.669899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:55.671196 (+ 1297us) server_negotiation.cc:407] Connection header received
0924 16:01:55.672462 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:55.672465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:55.672528 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:55.672614 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:55.674014 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.674695 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.676017 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.676326 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.676478 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:55.679134 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:55.679137 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:55.679139 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:55.679164 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:55.680865 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:55.681403 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:55.681407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:55.681411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:55.681510 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:55.681892 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:55.681897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:55.681899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:55.682081 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:55.682194 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:55.682684 (+ 490us) server_negotiation.cc:299] Negotiation successful
0924 16:01:55.682798 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:01:55.683700 27400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:55.670558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33761 (local address 127.25.109.2:46291)
0924 16:01:55.671039 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:55.671088 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:55.672213 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:55.672890 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:55.672897 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:55.673366 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:01:55.673861 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:55.673876 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.674820 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.674824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:55.675902 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:55.675918 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.676521 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.676525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:55.676737 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:55.677393 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:01:55.677413 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:01:55.678944 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:01:55.680974 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:55.680979 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:55.680989 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:55.681301 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:55.681627 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:55.681630 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:55.681633 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:55.681809 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:55.682208 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:01:55.682215 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:01:55.682439 (+ 224us) client_negotiation.cc:769] Sending connection context
0924 16:01:55.682639 (+ 200us) client_negotiation.cc:240] Negotiation successful
0924 16:01:55.682928 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":142,"threads_started":1}
I20260924 16:01:55.684859 27510 heartbeater.cc:347] Connected to a master server at 127.25.109.62:33761
I20260924 16:01:55.684996 27510 heartbeater.cc:464] Registering TS with master...
I20260924 16:01:55.685541 27510 heartbeater.cc:511] Master 127.25.109.62:33761 requested a full tablet report, sending...
I20260924 16:01:55.686631 27161 ts_manager.cc:194] Registered new tserver with Master: 134b3b306d91453ebf1ed1314b21d3a8 (127.25.109.2:43127)
I20260924 16:01:55.687470 27161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.2@KRBTEST.COM'} at 127.25.109.2:46291
I20260924 16:01:55.687686 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 27376
I20260924 16:01:55.687821 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance
I20260924 16:01:55.688148 26036 external_mini_cluster.cc:1428] Setting key 0c93b3024975b05db35d2ff45620e720
I20260924 16:01:55.694789 27212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:55.687185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:60331 (local address 127.25.109.62:33761)
0924 16:01:55.687297 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:55.687300 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:55.687565 (+ 265us) server_negotiation.cc:407] Connection header received
0924 16:01:55.687673 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:55.687676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:55.687721 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:55.687799 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:55.688432 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.689010 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.690052 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.690282 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.690391 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:55.691666 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:55.691669 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:55.691671 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:55.691693 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:55.693109 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:55.693542 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:55.693546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:55.693548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:55.693628 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:55.693918 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:55.693921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:55.693923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:55.694138 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:55.694269 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:55.694526 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 16:01:55.694631 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 16:01:55.694824 27516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:55.687092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33761 (local address 127.25.109.2:60331)
0924 16:01:55.687454 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:55.687470 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:55.687576 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:55.687935 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:55.687938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:55.688132 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:01:55.688285 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:55.688293 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.689151 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.689154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:55.689924 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:55.689933 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.690406 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.690410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:55.690504 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:55.691047 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:01:55.691066 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:01:55.691521 (+ 455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:01:55.693230 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:55.693233 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:55.693236 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:55.693431 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:55.693726 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:55.693730 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:55.693732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:55.693817 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:55.694260 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:01:55.694262 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:01:55.694359 (+ 97us) client_negotiation.cc:769] Sending connection context
0924 16:01:55.694470 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 16:01:55.694622 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":3,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.25.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
I20260924 16:01:55.715977 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:0
--local_ip_for_outbound_sockets=127.25.109.3
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:33761
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:43535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:01:55.829654 27519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:01:55.829909 27519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:01:55.829953 27519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:01:55.834050 27519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:01:55.834105 27519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:01:55.834142 27519 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:01:55.834175 27519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:01:55.839694 27519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:43535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.109.62:33761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:01:55.840794 27519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:01:55.841897 27519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:01:55.851418 27524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:01:55.851378 27527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:01:55.851447 27525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:01:55.851617 27519 server_base.cc:1034] running on GCE node
I20260924 16:01:55.852164 27519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:01:55.852852 27519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:01:55.854046 27519 hybrid_clock.cc:648] HybridClock initialized: now 1790265715854030 us; error 27 us; skew 500 ppm
Sep 24 16:01:55 dist-test-slave-n106 krb5kdc[27117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265715, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:01:55.856890 27519 init.cc:377] Logged in from keytab as kudu/127.25.109.3@KRBTEST.COM (short username kudu)
I20260924 16:01:55.857860 27519 webserver.cc:460] Webserver started at http://127.25.109.3:38073/ using document root <none> and password file <none>
I20260924 16:01:55.858429 27519 fs_manager.cc:359] Metadata directory not provided
I20260924 16:01:55.858523 27519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:01:55.858757 27519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:01:55.860832 27519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-2/data/instance:
uuid: "3ec57096ec2041b59bf3457ba5712e01"
format_stamp: "Formatted at 2026-09-24 16:01:55 on dist-test-slave-n106"
server_key: "8c5f72cac3281ad6014ef4270f065758"
server_key_iv: "35d3b62161c0b814d824a91622145dfb"
server_key_version: "encryptionkey@0"
I20260924 16:01:55.861397 27519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance:
uuid: "3ec57096ec2041b59bf3457ba5712e01"
format_stamp: "Formatted at 2026-09-24 16:01:55 on dist-test-slave-n106"
server_key: "8c5f72cac3281ad6014ef4270f065758"
server_key_iv: "35d3b62161c0b814d824a91622145dfb"
server_key_version: "encryptionkey@0"
I20260924 16:01:55.865123 27519 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:01:55.867558 27534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:55.868865 27519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:01:55.869010 27519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "3ec57096ec2041b59bf3457ba5712e01"
format_stamp: "Formatted at 2026-09-24 16:01:55 on dist-test-slave-n106"
server_key: "8c5f72cac3281ad6014ef4270f065758"
server_key_iv: "35d3b62161c0b814d824a91622145dfb"
server_key_version: "encryptionkey@0"
I20260924 16:01:55.869131 27519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:01:55.893756 27519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:01:55.897356 27519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:01:55.897589 27519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:01:55.899222 27519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:01:55.899304 27519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:55.899381 27519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:01:55.899427 27519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:01:55 dist-test-slave-n106 krb5kdc[27117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265715, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:01:55.910974 27519 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:42955
I20260924 16:01:55.911206 27651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:42955 every 8 connection(s)
I20260924 16:01:55.912045 27519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:01:55.912539 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 27519
I20260924 16:01:55.912676 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance
I20260924 16:01:55.913018 26036 external_mini_cluster.cc:1428] Setting key a67558e0e90230fc2b64de0d252c7d72
I20260924 16:01:55.914372 27212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:55.900133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:33185 (local address 127.25.109.62:33761)
0924 16:01:55.900266 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:55.900270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:55.901400 (+ 1130us) server_negotiation.cc:407] Connection header received
0924 16:01:55.902617 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:55.902620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:55.902708 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:55.902797 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:55.904524 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.905433 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.906621 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.907024 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.907154 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:55.910062 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:55.910066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:55.910069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:55.910100 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:55.911668 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:55.912292 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:55.912298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:55.912299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:55.912480 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:55.913070 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:55.913072 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:55.913074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:55.913259 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:55.913383 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:55.914080 (+ 697us) server_negotiation.cc:299] Negotiation successful
0924 16:01:55.914189 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 16:01:55.915307 27543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:55.900723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33761 (local address 127.25.109.3:33185)
0924 16:01:55.901235 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:55.901279 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:55.902358 (+ 1079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:55.903040 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:55.903050 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:55.903571 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:01:55.904274 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:55.904290 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.905588 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.905592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:55.906485 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:55.906495 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.907212 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.907217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:55.907403 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:55.908027 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:01:55.908046 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:01:55.909882 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:01:55.911794 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:55.911800 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:55.911814 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:55.912177 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:55.912611 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:55.912615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:55.912618 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:55.912806 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:55.913457 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:01:55.913466 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:01:55.913772 (+ 306us) client_negotiation.cc:769] Sending connection context
0924 16:01:55.913990 (+ 218us) client_negotiation.cc:240] Negotiation successful
0924 16:01:55.914310 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":148,"threads_started":1}
I20260924 16:01:55.916522 27652 heartbeater.cc:347] Connected to a master server at 127.25.109.62:33761
I20260924 16:01:55.916695 27652 heartbeater.cc:464] Registering TS with master...
I20260924 16:01:55.917255 27652 heartbeater.cc:511] Master 127.25.109.62:33761 requested a full tablet report, sending...
I20260924 16:01:55.918272 27161 ts_manager.cc:194] Registered new tserver with Master: 3ec57096ec2041b59bf3457ba5712e01 (127.25.109.3:42955)
I20260924 16:01:55.919064 27161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.3@KRBTEST.COM'} at 127.25.109.3:33185
I20260924 16:01:55.925834 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:55.918507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33761 (local address 127.25.109.3:47593)
0924 16:01:55.918813 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:55.918828 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:55.918921 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:55.919144 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:55.919150 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:55.919353 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:01:55.919479 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:55.919485 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.920240 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.920243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:55.921012 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:55.921023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.921545 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.921549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:55.921641 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:55.922176 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:01:55.922192 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:01:55.922506 (+ 314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:01:55.924281 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:55.924287 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:55.924289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:55.924565 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:55.924882 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:55.924884 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:55.924885 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:55.924971 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:55.925293 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:01:55.925295 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:01:55.925366 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 16:01:55.925517 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 16:01:55.925655 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":99,"threads_started":1}
I20260924 16:01:55.925848 27212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:55.918618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:47593 (local address 127.25.109.62:33761)
0924 16:01:55.918736 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:55.918740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:55.918881 (+ 141us) server_negotiation.cc:407] Connection header received
0924 16:01:55.918997 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:55.918999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:55.919033 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:55.919094 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:55.919600 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.920101 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.921155 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.921412 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.921544 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:55.922648 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:55.922652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:55.922654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:55.922679 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:55.924161 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:55.924688 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:55.924692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:55.924694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:55.924786 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:55.925054 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:55.925056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:55.925057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:55.925209 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:55.925286 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:55.925568 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 16:01:55.925685 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 16:01:55.927011 26036 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 16:01:55 dist-test-slave-n106 krb5kdc[27117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265715, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 16:01:55 dist-test-slave-n106 krb5kdc[27117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265715, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:01:55.952917 27212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:55.944861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48708 (local address 127.25.109.62:33761)
0924 16:01:55.945008 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:55.945012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:55.945024 (+ 12us) server_negotiation.cc:407] Connection header received
0924 16:01:55.945064 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:55.945066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:55.945104 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:55.945170 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:55.945853 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.946365 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.947158 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.947375 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.947466 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:55.949937 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:55.949941 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:55.949943 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:55.949968 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:55.951269 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:55.951654 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:55.951657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:55.951658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:55.951734 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:55.951980 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:55.951983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:55.951984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:55.952156 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:55.952259 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:55.952566 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 16:01:55.952710 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 16:01:55.956208 27161 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:01:55.959379 27161 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:01:55.974380 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:55.967912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:43127 (local address 127.0.0.1:39172)
0924 16:01:55.969041 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:55.969092 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:55.969232 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:55.969849 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:55.969855 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:55.969877 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:01:55.970010 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:55.970016 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.971220 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.971226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:55.972695 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:55.972712 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.973833 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.973837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:55.973946 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:55.973986 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 16:01:55.974088 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 16:01:55.974206 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":940,"thread_start_us":90,"threads_started":1}
I20260924 16:01:55.974485 27672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:55.968064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39172 (local address 127.25.109.2:43127)
0924 16:01:55.968681 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:55.968695 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:55.969220 (+ 525us) server_negotiation.cc:407] Connection header received
0924 16:01:55.969460 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:55.969468 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:55.969688 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:55.969860 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:01:55.970109 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.971124 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.972824 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.973710 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.973815 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:55.974085 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:55.974144 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 16:01:55.974252 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":123,"threads_started":1}
I20260924 16:01:55.974996 27670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:55.968233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:42955 (local address 127.0.0.1:39218)
0924 16:01:55.969282 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:55.969297 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:55.969383 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:55.970283 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:55.970285 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:55.970304 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:01:55.970459 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:55.970466 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.971857 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.971860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:55.973164 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:55.973176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.974548 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.974551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:55.974664 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:55.974691 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 16:01:55.974766 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 16:01:55.974845 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"mutex_wait_us":60,"thread_start_us":175,"threads_started":1}
I20260924 16:01:55.975176 27674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:55.968791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41950 (local address 127.25.109.1:40269)
0924 16:01:55.969424 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:55.969439 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:55.969461 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:01:55.969691 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:55.969698 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:55.969886 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:55.970035 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:01:55.970634 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.971891 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.973479 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.974235 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.974330 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:55.974469 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:55.974930 (+ 461us) server_negotiation.cc:299] Negotiation successful
0924 16:01:55.975032 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":89,"threads_started":1}
I20260924 16:01:55.975458 27671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:55.967942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:40269 (local address 127.0.0.1:41950)
0924 16:01:55.969159 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:55.969171 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:55.969236 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:55.970185 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:55.970192 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:55.970214 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:01:55.970482 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:55.970491 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.972003 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.972006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:55.973347 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:55.973362 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.974541 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.974544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:55.974638 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:55.974661 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 16:01:55.974728 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 16:01:55.974803 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1134,"mutex_wait_us":93,"thread_start_us":108,"threads_started":1}
I20260924 16:01:55.975678 27675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:55.968836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39218 (local address 127.25.109.3:42955)
0924 16:01:55.969824 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:55.969839 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:55.969861 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:01:55.969930 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:55.969937 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:55.970139 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:55.970285 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:01:55.970579 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.971736 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.973304 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:55.974163 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:55.974265 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:55.974429 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:55.975418 (+ 989us) server_negotiation.cc:299] Negotiation successful
0924 16:01:55.975516 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":895,"thread_start_us":132,"threads_started":1}
I20260924 16:01:55.977890 27300 tablet_service.cc:1467] Processing CreateTablet for tablet a090803da8fd4cd2b4794d685a78b596 (DEFAULT_TABLE table=test-table [id=02343f1c1f1e41e3a209558d2d689a1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:01:55.977916 27442 tablet_service.cc:1467] Processing CreateTablet for tablet a090803da8fd4cd2b4794d685a78b596 (DEFAULT_TABLE table=test-table [id=02343f1c1f1e41e3a209558d2d689a1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:01:55.978260 27584 tablet_service.cc:1467] Processing CreateTablet for tablet a090803da8fd4cd2b4794d685a78b596 (DEFAULT_TABLE table=test-table [id=02343f1c1f1e41e3a209558d2d689a1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:01:55.978968 27442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a090803da8fd4cd2b4794d685a78b596. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:01:55.978968 27300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a090803da8fd4cd2b4794d685a78b596. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:01:55.979075 27584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a090803da8fd4cd2b4794d685a78b596. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:01:55.985303 27676 tablet_bootstrap.cc:492] T a090803da8fd4cd2b4794d685a78b596 P 3df6aff1795d498bbfbd1046d78be724: Bootstrap starting.
I20260924 16:01:55.987083 27677 tablet_bootstrap.cc:492] T a090803da8fd4cd2b4794d685a78b596 P 3ec57096ec2041b59bf3457ba5712e01: Bootstrap starting.
I20260924 16:01:55.987511 27678 tablet_bootstrap.cc:492] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8: Bootstrap starting.
I20260924 16:01:55.987744 27676 tablet_bootstrap.cc:654] T a090803da8fd4cd2b4794d685a78b596 P 3df6aff1795d498bbfbd1046d78be724: Neither blocks nor log segments found. Creating new log.
I20260924 16:01:55.988924 27676 log.cc:824] T a090803da8fd4cd2b4794d685a78b596 P 3df6aff1795d498bbfbd1046d78be724: Log is configured to *not* fsync() on all Append() calls
I20260924 16:01:55.989198 27677 tablet_bootstrap.cc:654] T a090803da8fd4cd2b4794d685a78b596 P 3ec57096ec2041b59bf3457ba5712e01: Neither blocks nor log segments found. Creating new log.
I20260924 16:01:55.989501 27678 tablet_bootstrap.cc:654] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8: Neither blocks nor log segments found. Creating new log.
I20260924 16:01:55.990067 27677 log.cc:824] T a090803da8fd4cd2b4794d685a78b596 P 3ec57096ec2041b59bf3457ba5712e01: Log is configured to *not* fsync() on all Append() calls
I20260924 16:01:55.990378 27678 log.cc:824] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8: Log is configured to *not* fsync() on all Append() calls
I20260924 16:01:55.991394 27676 tablet_bootstrap.cc:492] T a090803da8fd4cd2b4794d685a78b596 P 3df6aff1795d498bbfbd1046d78be724: No bootstrap required, opened a new log
I20260924 16:01:55.991590 27676 ts_tablet_manager.cc:1397] T a090803da8fd4cd2b4794d685a78b596 P 3df6aff1795d498bbfbd1046d78be724: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 16:01:55.991995 27677 tablet_bootstrap.cc:492] T a090803da8fd4cd2b4794d685a78b596 P 3ec57096ec2041b59bf3457ba5712e01: No bootstrap required, opened a new log
I20260924 16:01:55.992201 27677 ts_tablet_manager.cc:1397] T a090803da8fd4cd2b4794d685a78b596 P 3ec57096ec2041b59bf3457ba5712e01: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 16:01:55.992328 27678 tablet_bootstrap.cc:492] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8: No bootstrap required, opened a new log
I20260924 16:01:55.992539 27678 ts_tablet_manager.cc:1397] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 16:01:55.995030 27677 raft_consensus.cc:348] T a090803da8fd4cd2b4794d685a78b596 P 3ec57096ec2041b59bf3457ba5712e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } }
I20260924 16:01:55.995272 27677 raft_consensus.cc:374] T a090803da8fd4cd2b4794d685a78b596 P 3ec57096ec2041b59bf3457ba5712e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:01:55.995353 27677 raft_consensus.cc:729] T a090803da8fd4cd2b4794d685a78b596 P 3ec57096ec2041b59bf3457ba5712e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ec57096ec2041b59bf3457ba5712e01, State: Initialized, Role: FOLLOWER
I20260924 16:01:55.995536 27676 raft_consensus.cc:348] T a090803da8fd4cd2b4794d685a78b596 P 3df6aff1795d498bbfbd1046d78be724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } }
I20260924 16:01:55.995894 27676 raft_consensus.cc:374] T a090803da8fd4cd2b4794d685a78b596 P 3df6aff1795d498bbfbd1046d78be724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:01:55.995877 27677 consensus_queue.cc:260] T a090803da8fd4cd2b4794d685a78b596 P 3ec57096ec2041b59bf3457ba5712e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } }
I20260924 16:01:55.995996 27676 raft_consensus.cc:729] T a090803da8fd4cd2b4794d685a78b596 P 3df6aff1795d498bbfbd1046d78be724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3df6aff1795d498bbfbd1046d78be724, State: Initialized, Role: FOLLOWER
I20260924 16:01:55.996548 27676 consensus_queue.cc:260] T a090803da8fd4cd2b4794d685a78b596 P 3df6aff1795d498bbfbd1046d78be724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } }
I20260924 16:01:55.996630 27678 raft_consensus.cc:348] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } }
I20260924 16:01:55.996940 27678 raft_consensus.cc:374] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:01:55.997030 27678 raft_consensus.cc:729] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 134b3b306d91453ebf1ed1314b21d3a8, State: Initialized, Role: FOLLOWER
I20260924 16:01:55.997220 27367 heartbeater.cc:503] Master 127.25.109.62:33761 was elected leader, sending a full tablet report...
I20260924 16:01:55.997506 27676 ts_tablet_manager.cc:1428] T a090803da8fd4cd2b4794d685a78b596 P 3df6aff1795d498bbfbd1046d78be724: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 16:01:55.997468 27678 consensus_queue.cc:260] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } }
I20260924 16:01:55.997067 27677 ts_tablet_manager.cc:1428] T a090803da8fd4cd2b4794d685a78b596 P 3ec57096ec2041b59bf3457ba5712e01: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260924 16:01:55.998190 27652 heartbeater.cc:503] Master 127.25.109.62:33761 was elected leader, sending a full tablet report...
I20260924 16:01:55.998520 27678 ts_tablet_manager.cc:1428] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 16:01:55.998612 27510 heartbeater.cc:503] Master 127.25.109.62:33761 was elected leader, sending a full tablet report...
I20260924 16:01:56.063073 27684 raft_consensus.cc:482] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:01:56.063377 27684 raft_consensus.cc:504] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } }
I20260924 16:01:56.064854 27684 leader_election.cc:288] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3df6aff1795d498bbfbd1046d78be724 (127.25.109.1:40269), 3ec57096ec2041b59bf3457ba5712e01 (127.25.109.3:42955)
I20260924 16:01:56.066956 27683 raft_consensus.cc:482] T a090803da8fd4cd2b4794d685a78b596 P 3df6aff1795d498bbfbd1046d78be724 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:01:56.067137 27683 raft_consensus.cc:504] T a090803da8fd4cd2b4794d685a78b596 P 3df6aff1795d498bbfbd1046d78be724 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } }
I20260924 16:01:56.068154 27683 leader_election.cc:288] T a090803da8fd4cd2b4794d685a78b596 P 3df6aff1795d498bbfbd1046d78be724 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 134b3b306d91453ebf1ed1314b21d3a8 (127.25.109.2:43127), 3ec57096ec2041b59bf3457ba5712e01 (127.25.109.3:42955)
I20260924 16:01:56.069718 27400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:56.065052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:40269 (local address 127.25.109.2:48505)
0924 16:01:56.065318 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:56.065334 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:56.065435 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:56.065754 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:56.065757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:56.065774 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:01:56.065898 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:56.065904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.066738 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.066742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:56.068227 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:56.068238 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.069276 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.069280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:56.069392 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:56.069415 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 16:01:56.069482 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 16:01:56.069557 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260924 16:01:56.070384 27674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:56.065143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:48505 (local address 127.25.109.1:40269)
0924 16:01:56.065307 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:56.065312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:56.065406 (+ 94us) server_negotiation.cc:407] Connection header received
0924 16:01:56.065586 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:56.065592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:56.065653 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:56.065742 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:01:56.066016 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.066604 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.068377 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.069042 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.069163 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:56.069219 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:56.070191 (+ 972us) server_negotiation.cc:299] Negotiation successful
0924 16:01:56.070272 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:01:56.070397 27675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:56.065383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:43347 (local address 127.25.109.3:42955)
0924 16:01:56.065526 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:56.065534 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:56.065574 (+ 40us) server_negotiation.cc:407] Connection header received
0924 16:01:56.065743 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:56.065747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:56.065797 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:56.065882 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:01:56.066167 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.066675 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.068932 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.069595 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.069682 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:56.069730 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:56.070207 (+ 477us) server_negotiation.cc:299] Negotiation successful
0924 16:01:56.070295 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:01:56.070894 27685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:56.065127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:42955 (local address 127.25.109.2:43347)
0924 16:01:56.065498 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:56.065512 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:56.065617 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:56.065893 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:56.065896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:56.065913 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:01:56.066043 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:56.066048 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.066800 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.066803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:56.068791 (+ 1988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:56.068801 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.069826 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.069832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:56.070049 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:56.070075 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 16:01:56.070567 (+ 492us) client_negotiation.cc:240] Negotiation successful
0924 16:01:56.070665 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":1,"thread_start_us":118,"threads_started":1}
I20260924 16:01:56.070948 27321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a090803da8fd4cd2b4794d685a78b596" candidate_uuid: "134b3b306d91453ebf1ed1314b21d3a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3df6aff1795d498bbfbd1046d78be724" is_pre_election: true
I20260924 16:01:56.071163 27321 raft_consensus.cc:2454] T a090803da8fd4cd2b4794d685a78b596 P 3df6aff1795d498bbfbd1046d78be724 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 134b3b306d91453ebf1ed1314b21d3a8 in term 0.
I20260924 16:01:56.071738 27605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a090803da8fd4cd2b4794d685a78b596" candidate_uuid: "134b3b306d91453ebf1ed1314b21d3a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ec57096ec2041b59bf3457ba5712e01" is_pre_election: true
I20260924 16:01:56.072021 27605 raft_consensus.cc:2454] T a090803da8fd4cd2b4794d685a78b596 P 3ec57096ec2041b59bf3457ba5712e01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 134b3b306d91453ebf1ed1314b21d3a8 in term 0.
I20260924 16:01:56.072194 27395 leader_election.cc:302] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 134b3b306d91453ebf1ed1314b21d3a8, 3df6aff1795d498bbfbd1046d78be724; no voters:
I20260924 16:01:56.072621 27684 raft_consensus.cc:2781] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:01:56.072811 27684 raft_consensus.cc:482] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:01:56.072903 27684 raft_consensus.cc:3037] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:01:56.074693 27688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:56.068750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:49617 (local address 127.25.109.3:42955)
0924 16:01:56.069653 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:56.069685 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:56.069698 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:01:56.069736 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:56.069739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:56.069797 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:56.069921 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:01:56.070428 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.071408 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.073289 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.073943 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.074079 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:56.074137 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:56.074454 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 16:01:56.074529 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":95,"threads_started":1}
I20260924 16:01:56.074904 27687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:56.068450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:42955 (local address 127.25.109.1:49617)
0924 16:01:56.069165 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:56.069179 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:56.069284 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:56.070025 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:56.070029 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:56.070051 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:01:56.070297 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:56.070305 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.071564 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.071568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:56.073179 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:56.073189 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.074072 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.074076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:56.074238 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:56.074263 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 16:01:56.074413 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 16:01:56.074505 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"mutex_wait_us":82,"thread_start_us":70,"threads_started":1}
I20260924 16:01:56.074988 27672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:56.068702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:45385 (local address 127.25.109.2:43127)
0924 16:01:56.068802 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:56.068807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:56.069151 (+ 344us) server_negotiation.cc:407] Connection header received
0924 16:01:56.069364 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:56.069367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:56.069425 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:56.069485 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:01:56.070066 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.070921 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.073284 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.074033 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.074191 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:56.074252 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:56.074721 (+ 469us) server_negotiation.cc:299] Negotiation successful
0924 16:01:56.074792 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":40}
I20260924 16:01:56.075146 27686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:56.068302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:43127 (local address 127.25.109.1:45385)
0924 16:01:56.069043 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:56.069071 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:56.069242 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:56.069641 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:56.069645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:56.069666 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:01:56.069937 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:56.069946 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.071066 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.071069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:56.073128 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:56.073140 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.074238 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.074242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:56.074504 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:56.074525 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 16:01:56.074909 (+ 384us) client_negotiation.cc:240] Negotiation successful
0924 16:01:56.074990 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":106,"thread_start_us":160,"threads_started":1}
I20260924 16:01:56.074870 27684 raft_consensus.cc:504] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } }
I20260924 16:01:56.075527 27605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a090803da8fd4cd2b4794d685a78b596" candidate_uuid: "3df6aff1795d498bbfbd1046d78be724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ec57096ec2041b59bf3457ba5712e01" is_pre_election: true
I20260924 16:01:56.075570 27684 leader_election.cc:288] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8 [CANDIDATE]: Term 1 election: Requested vote from peers 3df6aff1795d498bbfbd1046d78be724 (127.25.109.1:40269), 3ec57096ec2041b59bf3457ba5712e01 (127.25.109.3:42955)
I20260924 16:01:56.075672 27605 raft_consensus.cc:2454] T a090803da8fd4cd2b4794d685a78b596 P 3ec57096ec2041b59bf3457ba5712e01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3df6aff1795d498bbfbd1046d78be724 in term 0.
I20260924 16:01:56.075820 27463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a090803da8fd4cd2b4794d685a78b596" candidate_uuid: "3df6aff1795d498bbfbd1046d78be724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "134b3b306d91453ebf1ed1314b21d3a8" is_pre_election: true
I20260924 16:01:56.075917 27321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a090803da8fd4cd2b4794d685a78b596" candidate_uuid: "134b3b306d91453ebf1ed1314b21d3a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3df6aff1795d498bbfbd1046d78be724"
I20260924 16:01:56.076052 27321 raft_consensus.cc:3037] T a090803da8fd4cd2b4794d685a78b596 P 3df6aff1795d498bbfbd1046d78be724 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:01:56.076045 27463 raft_consensus.cc:2379] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3df6aff1795d498bbfbd1046d78be724 in current term 1: Already voted for candidate 134b3b306d91453ebf1ed1314b21d3a8 in this term.
I20260924 16:01:56.076114 27252 leader_election.cc:302] T a090803da8fd4cd2b4794d685a78b596 P 3df6aff1795d498bbfbd1046d78be724 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3df6aff1795d498bbfbd1046d78be724, 3ec57096ec2041b59bf3457ba5712e01; no voters:
I20260924 16:01:56.077590 27604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a090803da8fd4cd2b4794d685a78b596" candidate_uuid: "134b3b306d91453ebf1ed1314b21d3a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ec57096ec2041b59bf3457ba5712e01"
I20260924 16:01:56.077749 27321 raft_consensus.cc:2454] T a090803da8fd4cd2b4794d685a78b596 P 3df6aff1795d498bbfbd1046d78be724 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 134b3b306d91453ebf1ed1314b21d3a8 in term 1.
I20260924 16:01:56.077733 27604 raft_consensus.cc:3037] T a090803da8fd4cd2b4794d685a78b596 P 3ec57096ec2041b59bf3457ba5712e01 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:01:56.077891 27683 raft_consensus.cc:2741] T a090803da8fd4cd2b4794d685a78b596 P 3df6aff1795d498bbfbd1046d78be724 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260924 16:01:56.078116 27395 leader_election.cc:302] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 134b3b306d91453ebf1ed1314b21d3a8, 3df6aff1795d498bbfbd1046d78be724; no voters:
I20260924 16:01:56.078374 27684 raft_consensus.cc:2781] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:01:56.078671 27684 raft_consensus.cc:686] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8 [term 1 LEADER]: Becoming Leader. State: Replica: 134b3b306d91453ebf1ed1314b21d3a8, State: Running, Role: LEADER
I20260924 16:01:56.079118 27684 consensus_queue.cc:237] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } }
I20260924 16:01:56.079721 27604 raft_consensus.cc:2454] T a090803da8fd4cd2b4794d685a78b596 P 3ec57096ec2041b59bf3457ba5712e01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 134b3b306d91453ebf1ed1314b21d3a8 in term 1.
I20260924 16:01:56.082474 27160 catalog_manager.cc:5495] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 134b3b306d91453ebf1ed1314b21d3a8 (127.25.109.2). New cstate: current_term: 1 leader_uuid: "134b3b306d91453ebf1ed1314b21d3a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } health_report { overall_health: UNKNOWN } } }
I20260924 16:01:56.100469 27672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:56.096500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39182 (local address 127.25.109.2:43127)
0924 16:01:56.096641 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:56.096648 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:56.096806 (+ 158us) server_negotiation.cc:407] Connection header received
0924 16:01:56.097091 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:56.097093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:56.097146 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:56.097256 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:01:56.097534 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.098029 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.098915 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.099135 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.099245 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:56.099610 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:01:56.099999 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:01:56.100127 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:56.100207 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 16:01:56.100266 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 16:01:56.106426 27321 raft_consensus.cc:1261] T a090803da8fd4cd2b4794d685a78b596 P 3df6aff1795d498bbfbd1046d78be724 [term 1 FOLLOWER]: Refusing update from remote peer 134b3b306d91453ebf1ed1314b21d3a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:01:56.106426 27604 raft_consensus.cc:1261] T a090803da8fd4cd2b4794d685a78b596 P 3ec57096ec2041b59bf3457ba5712e01 [term 1 FOLLOWER]: Refusing update from remote peer 134b3b306d91453ebf1ed1314b21d3a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:01:56.107190 27684 consensus_queue.cc:1035] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:01:56.107437 27689 consensus_queue.cc:1035] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:01:56.115877 27692 mvcc.cc:205] Tried to move back new op lower bound from 7332928373164187648 to 7332928373063487488. Current Snapshot: MvccSnapshot[applied={T|T < 7332928373164187648}]
I20260924 16:01:56.116151 27693 mvcc.cc:205] Tried to move back new op lower bound from 7332928373164187648 to 7332928373063487488. Current Snapshot: MvccSnapshot[applied={T|T < 7332928373164187648}]
I20260924 16:01:56.129565 27160 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:01:56.129731 27160 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:01:56.132339 27160 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 973b347f0b28409f991e362af9865bfe: Sending DeleteTablet for 3 replicas of tablet a090803da8fd4cd2b4794d685a78b596
I20260924 16:01:56.133212 27442 tablet_service.cc:1514] Processing DeleteTablet for tablet a090803da8fd4cd2b4794d685a78b596 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:01:56 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:39172
I20260924 16:01:56.133260 27584 tablet_service.cc:1514] Processing DeleteTablet for tablet a090803da8fd4cd2b4794d685a78b596 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:01:56 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:39218
I20260924 16:01:56.133388 27300 tablet_service.cc:1514] Processing DeleteTablet for tablet a090803da8fd4cd2b4794d685a78b596 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:01:56 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:41950
I20260924 16:01:56.133852 27703 tablet_replica.cc:330] T a090803da8fd4cd2b4794d685a78b596 P 3df6aff1795d498bbfbd1046d78be724: stopping tablet replica
I20260924 16:01:56.133926 27702 tablet_replica.cc:330] T a090803da8fd4cd2b4794d685a78b596 P 3ec57096ec2041b59bf3457ba5712e01: stopping tablet replica
I20260924 16:01:56.134195 27702 raft_consensus.cc:2229] T a090803da8fd4cd2b4794d685a78b596 P 3ec57096ec2041b59bf3457ba5712e01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:01:56.134196 27703 raft_consensus.cc:2229] T a090803da8fd4cd2b4794d685a78b596 P 3df6aff1795d498bbfbd1046d78be724 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:01:56.134509 27702 raft_consensus.cc:2258] T a090803da8fd4cd2b4794d685a78b596 P 3ec57096ec2041b59bf3457ba5712e01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:01:56.134521 27703 raft_consensus.cc:2258] T a090803da8fd4cd2b4794d685a78b596 P 3df6aff1795d498bbfbd1046d78be724 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:01:56.134882 27701 tablet_replica.cc:330] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8: stopping tablet replica
I20260924 16:01:56.135043 27160 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 16:01:56.135181 27701 raft_consensus.cc:2229] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8 [term 1 LEADER]: Raft consensus shutting down.
W20260924 16:01:56.135475 27160 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:01:56.135572 27701 raft_consensus.cc:2258] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:01:56.136929 27703 ts_tablet_manager.cc:1905] T a090803da8fd4cd2b4794d685a78b596 P 3df6aff1795d498bbfbd1046d78be724: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:01:56.137326 27702 ts_tablet_manager.cc:1905] T a090803da8fd4cd2b4794d685a78b596 P 3ec57096ec2041b59bf3457ba5712e01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:01:56.140309 27703 ts_tablet_manager.cc:1918] T a090803da8fd4cd2b4794d685a78b596 P 3df6aff1795d498bbfbd1046d78be724: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:01:56.140491 27703 log.cc:1196] T a090803da8fd4cd2b4794d685a78b596 P 3df6aff1795d498bbfbd1046d78be724: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-0/wal/wals/a090803da8fd4cd2b4794d685a78b596
I20260924 16:01:56.140872 27703 ts_tablet_manager.cc:1939] T a090803da8fd4cd2b4794d685a78b596 P 3df6aff1795d498bbfbd1046d78be724: Deleting consensus metadata
I20260924 16:01:56.141342 27701 ts_tablet_manager.cc:1905] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:01:56.140369 27702 ts_tablet_manager.cc:1918] T a090803da8fd4cd2b4794d685a78b596 P 3ec57096ec2041b59bf3457ba5712e01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:01:56.142277 27702 log.cc:1196] T a090803da8fd4cd2b4794d685a78b596 P 3ec57096ec2041b59bf3457ba5712e01: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-2/wal/wals/a090803da8fd4cd2b4794d685a78b596
I20260924 16:01:56.142390 27442 tablet_service.cc:1467] Processing CreateTablet for tablet 9d195fcd041f4540ac8db1f5fedff170 (DEFAULT_TABLE table=test-table [id=f6ba1cd4404844ec9ae5ef9eb4948dbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:01:56.142669 27442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d195fcd041f4540ac8db1f5fedff170. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:01:56.142694 27702 ts_tablet_manager.cc:1939] T a090803da8fd4cd2b4794d685a78b596 P 3ec57096ec2041b59bf3457ba5712e01: Deleting consensus metadata
I20260924 16:01:56.144217 27701 ts_tablet_manager.cc:1918] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:01:56.144224 27584 tablet_service.cc:1467] Processing CreateTablet for tablet 9d195fcd041f4540ac8db1f5fedff170 (DEFAULT_TABLE table=test-table [id=f6ba1cd4404844ec9ae5ef9eb4948dbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:01:56.144301 27701 log.cc:1196] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-1/wal/wals/a090803da8fd4cd2b4794d685a78b596
I20260924 16:01:56.144479 27584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d195fcd041f4540ac8db1f5fedff170. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:01:56.144626 27701 ts_tablet_manager.cc:1939] T a090803da8fd4cd2b4794d685a78b596 P 134b3b306d91453ebf1ed1314b21d3a8: Deleting consensus metadata
I20260924 16:01:56.145195 27678 tablet_bootstrap.cc:492] T 9d195fcd041f4540ac8db1f5fedff170 P 134b3b306d91453ebf1ed1314b21d3a8: Bootstrap starting.
I20260924 16:01:56.146023 27678 tablet_bootstrap.cc:654] T 9d195fcd041f4540ac8db1f5fedff170 P 134b3b306d91453ebf1ed1314b21d3a8: Neither blocks nor log segments found. Creating new log.
I20260924 16:01:56.146518 27677 tablet_bootstrap.cc:492] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01: Bootstrap starting.
I20260924 16:01:56.146577 27147 catalog_manager.cc:4841] TS 3ec57096ec2041b59bf3457ba5712e01 (127.25.109.3:42955): tablet a090803da8fd4cd2b4794d685a78b596 (table test-table [id=02343f1c1f1e41e3a209558d2d689a1c]) successfully deleted
I20260924 16:01:56.146930 27149 catalog_manager.cc:4841] TS 3df6aff1795d498bbfbd1046d78be724 (127.25.109.1:40269): tablet a090803da8fd4cd2b4794d685a78b596 (table test-table [id=02343f1c1f1e41e3a209558d2d689a1c]) successfully deleted
I20260924 16:01:56.147194 27146 catalog_manager.cc:4841] TS 134b3b306d91453ebf1ed1314b21d3a8 (127.25.109.2:43127): tablet a090803da8fd4cd2b4794d685a78b596 (table test-table [id=02343f1c1f1e41e3a209558d2d689a1c]) successfully deleted
I20260924 16:01:56.147552 27677 tablet_bootstrap.cc:654] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01: Neither blocks nor log segments found. Creating new log.
I20260924 16:01:56.147572 27300 tablet_service.cc:1467] Processing CreateTablet for tablet 9d195fcd041f4540ac8db1f5fedff170 (DEFAULT_TABLE table=test-table [id=f6ba1cd4404844ec9ae5ef9eb4948dbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:01:56.147825 27300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d195fcd041f4540ac8db1f5fedff170. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:01:56.148610 27678 tablet_bootstrap.cc:492] T 9d195fcd041f4540ac8db1f5fedff170 P 134b3b306d91453ebf1ed1314b21d3a8: No bootstrap required, opened a new log
I20260924 16:01:56.148696 27678 ts_tablet_manager.cc:1397] T 9d195fcd041f4540ac8db1f5fedff170 P 134b3b306d91453ebf1ed1314b21d3a8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 16:01:56.149128 27678 raft_consensus.cc:348] T 9d195fcd041f4540ac8db1f5fedff170 P 134b3b306d91453ebf1ed1314b21d3a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } }
I20260924 16:01:56.149242 27678 raft_consensus.cc:374] T 9d195fcd041f4540ac8db1f5fedff170 P 134b3b306d91453ebf1ed1314b21d3a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:01:56.149272 27678 raft_consensus.cc:729] T 9d195fcd041f4540ac8db1f5fedff170 P 134b3b306d91453ebf1ed1314b21d3a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 134b3b306d91453ebf1ed1314b21d3a8, State: Initialized, Role: FOLLOWER
I20260924 16:01:56.149389 27678 consensus_queue.cc:260] T 9d195fcd041f4540ac8db1f5fedff170 P 134b3b306d91453ebf1ed1314b21d3a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } }
I20260924 16:01:56.149662 27678 ts_tablet_manager.cc:1428] T 9d195fcd041f4540ac8db1f5fedff170 P 134b3b306d91453ebf1ed1314b21d3a8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:01:56.150220 27677 tablet_bootstrap.cc:492] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01: No bootstrap required, opened a new log
I20260924 16:01:56.150297 27677 ts_tablet_manager.cc:1397] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 16:01:56.150672 27676 tablet_bootstrap.cc:492] T 9d195fcd041f4540ac8db1f5fedff170 P 3df6aff1795d498bbfbd1046d78be724: Bootstrap starting.
I20260924 16:01:56.150708 27677 raft_consensus.cc:348] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } }
I20260924 16:01:56.150805 27677 raft_consensus.cc:374] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:01:56.150846 27677 raft_consensus.cc:729] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ec57096ec2041b59bf3457ba5712e01, State: Initialized, Role: FOLLOWER
I20260924 16:01:56.150961 27677 consensus_queue.cc:260] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } }
I20260924 16:01:56.151199 27677 ts_tablet_manager.cc:1428] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:01:56.151664 27676 tablet_bootstrap.cc:654] T 9d195fcd041f4540ac8db1f5fedff170 P 3df6aff1795d498bbfbd1046d78be724: Neither blocks nor log segments found. Creating new log.
I20260924 16:01:56.152935 27676 tablet_bootstrap.cc:492] T 9d195fcd041f4540ac8db1f5fedff170 P 3df6aff1795d498bbfbd1046d78be724: No bootstrap required, opened a new log
I20260924 16:01:56.153021 27676 ts_tablet_manager.cc:1397] T 9d195fcd041f4540ac8db1f5fedff170 P 3df6aff1795d498bbfbd1046d78be724: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260924 16:01:56.153468 27676 raft_consensus.cc:348] T 9d195fcd041f4540ac8db1f5fedff170 P 3df6aff1795d498bbfbd1046d78be724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } }
I20260924 16:01:56.153599 27676 raft_consensus.cc:374] T 9d195fcd041f4540ac8db1f5fedff170 P 3df6aff1795d498bbfbd1046d78be724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:01:56.153640 27676 raft_consensus.cc:729] T 9d195fcd041f4540ac8db1f5fedff170 P 3df6aff1795d498bbfbd1046d78be724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3df6aff1795d498bbfbd1046d78be724, State: Initialized, Role: FOLLOWER
I20260924 16:01:56.153777 27676 consensus_queue.cc:260] T 9d195fcd041f4540ac8db1f5fedff170 P 3df6aff1795d498bbfbd1046d78be724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } }
I20260924 16:01:56.154048 27676 ts_tablet_manager.cc:1428] T 9d195fcd041f4540ac8db1f5fedff170 P 3df6aff1795d498bbfbd1046d78be724: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:01:56.156761 27682 raft_consensus.cc:482] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:01:56.156862 27682 raft_consensus.cc:504] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } }
I20260924 16:01:56.158001 27682 leader_election.cc:288] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3df6aff1795d498bbfbd1046d78be724 (127.25.109.1:40269), 134b3b306d91453ebf1ed1314b21d3a8 (127.25.109.2:43127)
I20260924 16:01:56.162079 27543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:56.158183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:40269 (local address 127.25.109.3:58773)
0924 16:01:56.158306 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:56.158322 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:56.158410 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:56.158790 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:56.158793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:56.158810 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:01:56.158924 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:56.158931 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.159902 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.159905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:56.160822 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:56.160833 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.161472 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.161475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:56.161719 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:56.161744 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 16:01:56.161866 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 16:01:56.161957 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 16:01:56.162079 27674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:56.158453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:58773 (local address 127.25.109.1:40269)
0924 16:01:56.158579 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:56.158584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:56.158596 (+ 12us) server_negotiation.cc:407] Connection header received
0924 16:01:56.158642 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:56.158645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:56.158692 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:56.158778 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:01:56.159065 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.159798 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.160952 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.161350 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.161480 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:56.161534 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:56.161884 (+ 350us) server_negotiation.cc:299] Negotiation successful
0924 16:01:56.161957 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 16:01:56.162546 27321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d195fcd041f4540ac8db1f5fedff170" candidate_uuid: "3ec57096ec2041b59bf3457ba5712e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3df6aff1795d498bbfbd1046d78be724" is_pre_election: true
I20260924 16:01:56.162695 27321 raft_consensus.cc:2454] T 9d195fcd041f4540ac8db1f5fedff170 P 3df6aff1795d498bbfbd1046d78be724 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ec57096ec2041b59bf3457ba5712e01 in term 0.
I20260924 16:01:56.163087 27538 leader_election.cc:302] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3df6aff1795d498bbfbd1046d78be724, 3ec57096ec2041b59bf3457ba5712e01; no voters:
I20260924 16:01:56.163388 27682 raft_consensus.cc:2781] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:01:56.163512 27682 raft_consensus.cc:482] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:01:56.163563 27682 raft_consensus.cc:3037] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:01:56.163708 27672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:56.158589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:46785 (local address 127.25.109.2:43127)
0924 16:01:56.158724 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:56.158729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:56.159161 (+ 432us) server_negotiation.cc:407] Connection header received
0924 16:01:56.159329 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:56.159332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:56.159374 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:56.159417 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:01:56.159806 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.160576 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.162106 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.162833 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.162932 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:56.162977 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:56.163495 (+ 518us) server_negotiation.cc:299] Negotiation successful
0924 16:01:56.163572 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:01:56.164037 27707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:56.158291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:43127 (local address 127.25.109.3:46785)
0924 16:01:56.159085 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:56.159098 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:56.159220 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:56.159548 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:56.159550 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:56.159565 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:01:56.159687 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:56.159692 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.160722 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.160726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:56.161961 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:56.161973 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.163208 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.163212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:56.163324 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:56.163347 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 16:01:56.163828 (+ 481us) client_negotiation.cc:240] Negotiation successful
0924 16:01:56.163905 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":67,"threads_started":1}
W20260924 16:01:56.163396 27653 tablet.cc:2369] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:01:56.165091 27682 raft_consensus.cc:504] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } }
I20260924 16:01:56.165302 27463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d195fcd041f4540ac8db1f5fedff170" candidate_uuid: "3ec57096ec2041b59bf3457ba5712e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "134b3b306d91453ebf1ed1314b21d3a8" is_pre_election: true
I20260924 16:01:56.165441 27463 raft_consensus.cc:2454] T 9d195fcd041f4540ac8db1f5fedff170 P 134b3b306d91453ebf1ed1314b21d3a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ec57096ec2041b59bf3457ba5712e01 in term 0.
I20260924 16:01:56.165490 27682 leader_election.cc:288] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01 [CANDIDATE]: Term 1 election: Requested vote from peers 3df6aff1795d498bbfbd1046d78be724 (127.25.109.1:40269), 134b3b306d91453ebf1ed1314b21d3a8 (127.25.109.2:43127)
I20260924 16:01:56.165832 27463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d195fcd041f4540ac8db1f5fedff170" candidate_uuid: "3ec57096ec2041b59bf3457ba5712e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "134b3b306d91453ebf1ed1314b21d3a8"
I20260924 16:01:56.165935 27463 raft_consensus.cc:3037] T 9d195fcd041f4540ac8db1f5fedff170 P 134b3b306d91453ebf1ed1314b21d3a8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:01:56.166011 27321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d195fcd041f4540ac8db1f5fedff170" candidate_uuid: "3ec57096ec2041b59bf3457ba5712e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3df6aff1795d498bbfbd1046d78be724"
I20260924 16:01:56.166110 27321 raft_consensus.cc:3037] T 9d195fcd041f4540ac8db1f5fedff170 P 3df6aff1795d498bbfbd1046d78be724 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:01:56.167120 27463 raft_consensus.cc:2454] T 9d195fcd041f4540ac8db1f5fedff170 P 134b3b306d91453ebf1ed1314b21d3a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ec57096ec2041b59bf3457ba5712e01 in term 1.
I20260924 16:01:56.167120 27321 raft_consensus.cc:2454] T 9d195fcd041f4540ac8db1f5fedff170 P 3df6aff1795d498bbfbd1046d78be724 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ec57096ec2041b59bf3457ba5712e01 in term 1.
I20260924 16:01:56.167418 27535 leader_election.cc:302] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 134b3b306d91453ebf1ed1314b21d3a8, 3ec57096ec2041b59bf3457ba5712e01; no voters:
I20260924 16:01:56.167650 27682 raft_consensus.cc:2781] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:01:56.167918 27682 raft_consensus.cc:686] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01 [term 1 LEADER]: Becoming Leader. State: Replica: 3ec57096ec2041b59bf3457ba5712e01, State: Running, Role: LEADER
I20260924 16:01:56.168171 27682 consensus_queue.cc:237] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } }
I20260924 16:01:56.170796 27160 catalog_manager.cc:5495] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ec57096ec2041b59bf3457ba5712e01 (127.25.109.3). New cstate: current_term: 1 leader_uuid: "3ec57096ec2041b59bf3457ba5712e01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } health_report { overall_health: HEALTHY } } }
W20260924 16:01:56.182129 27511 tablet.cc:2369] T 9d195fcd041f4540ac8db1f5fedff170 P 134b3b306d91453ebf1ed1314b21d3a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 16:01:56 dist-test-slave-n106 krb5kdc[27117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265715, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:01:56.211802 27212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:56.202647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48714 (local address 127.25.109.62:33761)
0924 16:01:56.202755 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:56.202758 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:56.202854 (+ 96us) server_negotiation.cc:407] Connection header received
0924 16:01:56.202985 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:56.202987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:56.203028 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:56.203097 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:56.203925 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.204493 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.205624 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.205839 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.205931 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:56.208536 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:56.208539 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:56.208541 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:56.208562 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:56.210062 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:56.210519 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:56.210522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:56.210529 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:56.210635 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:56.210928 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:56.210931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:56.210932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:56.211101 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:56.211208 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:56.211501 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 16:01:56.211667 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":41}
I20260924 16:01:56.211802 27715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:56.202424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33761 (local address 127.0.0.1:48714)
0924 16:01:56.202751 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:56.202768 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:56.202892 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:56.203153 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:56.203156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:56.203688 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:01:56.203807 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:56.203815 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.204612 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.204615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:56.205508 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:56.205519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.205963 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.205967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:56.206059 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:56.206636 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:01:56.206652 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:01:56.208370 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:01:56.210167 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:56.210178 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:56.210185 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:56.210409 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:56.210741 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:56.210744 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:56.210746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:56.210831 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:56.211208 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:01:56.211216 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:01:56.211331 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 16:01:56.211452 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 16:01:56.211602 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":107,"threads_started":1}
I20260924 16:01:56.215957 27160 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:48714:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 16:01:56.216184 27368 tablet.cc:2369] T 9d195fcd041f4540ac8db1f5fedff170 P 3df6aff1795d498bbfbd1046d78be724: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:01:56.216876 27160 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:01:56.221791 27300 tablet_service.cc:1467] Processing CreateTablet for tablet 0b3e8728954c430288a520a891419dee (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3488e0b539c04d2c8f388a9e9694e41e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:01:56.221868 27442 tablet_service.cc:1467] Processing CreateTablet for tablet 0b3e8728954c430288a520a891419dee (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3488e0b539c04d2c8f388a9e9694e41e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:01:56.221992 27584 tablet_service.cc:1467] Processing CreateTablet for tablet 0b3e8728954c430288a520a891419dee (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3488e0b539c04d2c8f388a9e9694e41e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:01:56.222189 27300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b3e8728954c430288a520a891419dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:01:56.222261 27442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b3e8728954c430288a520a891419dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:01:56.222344 27584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b3e8728954c430288a520a891419dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:01:56.225132 27676 tablet_bootstrap.cc:492] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724: Bootstrap starting.
I20260924 16:01:56.226073 27676 tablet_bootstrap.cc:654] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724: Neither blocks nor log segments found. Creating new log.
I20260924 16:01:56.226735 27678 tablet_bootstrap.cc:492] T 0b3e8728954c430288a520a891419dee P 134b3b306d91453ebf1ed1314b21d3a8: Bootstrap starting.
I20260924 16:01:56.227619 27678 tablet_bootstrap.cc:654] T 0b3e8728954c430288a520a891419dee P 134b3b306d91453ebf1ed1314b21d3a8: Neither blocks nor log segments found. Creating new log.
I20260924 16:01:56.228235 27677 tablet_bootstrap.cc:492] T 0b3e8728954c430288a520a891419dee P 3ec57096ec2041b59bf3457ba5712e01: Bootstrap starting.
I20260924 16:01:56.229173 27677 tablet_bootstrap.cc:654] T 0b3e8728954c430288a520a891419dee P 3ec57096ec2041b59bf3457ba5712e01: Neither blocks nor log segments found. Creating new log.
I20260924 16:01:56.230142 27678 tablet_bootstrap.cc:492] T 0b3e8728954c430288a520a891419dee P 134b3b306d91453ebf1ed1314b21d3a8: No bootstrap required, opened a new log
I20260924 16:01:56.230234 27678 ts_tablet_manager.cc:1397] T 0b3e8728954c430288a520a891419dee P 134b3b306d91453ebf1ed1314b21d3a8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 16:01:56.230413 27677 tablet_bootstrap.cc:492] T 0b3e8728954c430288a520a891419dee P 3ec57096ec2041b59bf3457ba5712e01: No bootstrap required, opened a new log
I20260924 16:01:56.230530 27677 ts_tablet_manager.cc:1397] T 0b3e8728954c430288a520a891419dee P 3ec57096ec2041b59bf3457ba5712e01: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 16:01:56.230681 27678 raft_consensus.cc:348] T 0b3e8728954c430288a520a891419dee P 134b3b306d91453ebf1ed1314b21d3a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } }
I20260924 16:01:56.230806 27678 raft_consensus.cc:374] T 0b3e8728954c430288a520a891419dee P 134b3b306d91453ebf1ed1314b21d3a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:01:56.230839 27678 raft_consensus.cc:729] T 0b3e8728954c430288a520a891419dee P 134b3b306d91453ebf1ed1314b21d3a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 134b3b306d91453ebf1ed1314b21d3a8, State: Initialized, Role: FOLLOWER
I20260924 16:01:56.230973 27678 consensus_queue.cc:260] T 0b3e8728954c430288a520a891419dee P 134b3b306d91453ebf1ed1314b21d3a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } }
I20260924 16:01:56.231016 27677 raft_consensus.cc:348] T 0b3e8728954c430288a520a891419dee P 3ec57096ec2041b59bf3457ba5712e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } }
I20260924 16:01:56.231146 27677 raft_consensus.cc:374] T 0b3e8728954c430288a520a891419dee P 3ec57096ec2041b59bf3457ba5712e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:01:56.231184 27677 raft_consensus.cc:729] T 0b3e8728954c430288a520a891419dee P 3ec57096ec2041b59bf3457ba5712e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ec57096ec2041b59bf3457ba5712e01, State: Initialized, Role: FOLLOWER
I20260924 16:01:56.231231 27678 ts_tablet_manager.cc:1428] T 0b3e8728954c430288a520a891419dee P 134b3b306d91453ebf1ed1314b21d3a8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:01:56.231309 27677 consensus_queue.cc:260] T 0b3e8728954c430288a520a891419dee P 3ec57096ec2041b59bf3457ba5712e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } }
I20260924 16:01:56.231832 27676 tablet_bootstrap.cc:492] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724: No bootstrap required, opened a new log
I20260924 16:01:56.231801 27682 tablet_replica.cc:439] T 0b3e8728954c430288a520a891419dee P 3ec57096ec2041b59bf3457ba5712e01: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } } }
I20260924 16:01:56.231921 27676 ts_tablet_manager.cc:1397] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20260924 16:01:56.231928 27682 tablet_replica.cc:442] T 0b3e8728954c430288a520a891419dee P 3ec57096ec2041b59bf3457ba5712e01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:01:56.232155 27677 ts_tablet_manager.cc:1428] T 0b3e8728954c430288a520a891419dee P 3ec57096ec2041b59bf3457ba5712e01: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 16:01:56.232324 27676 raft_consensus.cc:348] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } }
I20260924 16:01:56.232465 27676 raft_consensus.cc:374] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:01:56.232497 27676 raft_consensus.cc:729] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3df6aff1795d498bbfbd1046d78be724, State: Initialized, Role: FOLLOWER
I20260924 16:01:56.232625 27676 consensus_queue.cc:260] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } }
I20260924 16:01:56.232864 27676 ts_tablet_manager.cc:1428] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:01:56.232861 27683 tablet_replica.cc:439] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } } }
I20260924 16:01:56.233086 27683 tablet_replica.cc:442] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:01:56.234294 27684 tablet_replica.cc:439] T 0b3e8728954c430288a520a891419dee P 134b3b306d91453ebf1ed1314b21d3a8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } } }
I20260924 16:01:56.234434 27684 tablet_replica.cc:442] T 0b3e8728954c430288a520a891419dee P 134b3b306d91453ebf1ed1314b21d3a8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:01:56.309211 27683 raft_consensus.cc:482] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:01:56.309366 27683 raft_consensus.cc:504] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } }
I20260924 16:01:56.309940 27683 leader_election.cc:288] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 134b3b306d91453ebf1ed1314b21d3a8 (127.25.109.2:43127), 3ec57096ec2041b59bf3457ba5712e01 (127.25.109.3:42955)
I20260924 16:01:56.310400 27463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b3e8728954c430288a520a891419dee" candidate_uuid: "3df6aff1795d498bbfbd1046d78be724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "134b3b306d91453ebf1ed1314b21d3a8" is_pre_election: true
I20260924 16:01:56.310374 27604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b3e8728954c430288a520a891419dee" candidate_uuid: "3df6aff1795d498bbfbd1046d78be724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ec57096ec2041b59bf3457ba5712e01" is_pre_election: true
I20260924 16:01:56.310582 27604 raft_consensus.cc:2454] T 0b3e8728954c430288a520a891419dee P 3ec57096ec2041b59bf3457ba5712e01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3df6aff1795d498bbfbd1046d78be724 in term 0.
I20260924 16:01:56.310571 27463 raft_consensus.cc:2454] T 0b3e8728954c430288a520a891419dee P 134b3b306d91453ebf1ed1314b21d3a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3df6aff1795d498bbfbd1046d78be724 in term 0.
I20260924 16:01:56.311053 27251 leader_election.cc:302] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 134b3b306d91453ebf1ed1314b21d3a8, 3df6aff1795d498bbfbd1046d78be724; no voters:
I20260924 16:01:56.311239 27683 raft_consensus.cc:2781] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:01:56.311311 27683 raft_consensus.cc:482] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:01:56.311362 27683 raft_consensus.cc:3037] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:01:56.314920 27683 raft_consensus.cc:504] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } }
I20260924 16:01:56.315351 27683 leader_election.cc:288] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724 [CANDIDATE]: Term 1 election: Requested vote from peers 134b3b306d91453ebf1ed1314b21d3a8 (127.25.109.2:43127), 3ec57096ec2041b59bf3457ba5712e01 (127.25.109.3:42955)
I20260924 16:01:56.315726 27463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b3e8728954c430288a520a891419dee" candidate_uuid: "3df6aff1795d498bbfbd1046d78be724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "134b3b306d91453ebf1ed1314b21d3a8"
I20260924 16:01:56.315894 27463 raft_consensus.cc:3037] T 0b3e8728954c430288a520a891419dee P 134b3b306d91453ebf1ed1314b21d3a8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:01:56.315874 27604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b3e8728954c430288a520a891419dee" candidate_uuid: "3df6aff1795d498bbfbd1046d78be724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ec57096ec2041b59bf3457ba5712e01"
I20260924 16:01:56.316002 27604 raft_consensus.cc:3037] T 0b3e8728954c430288a520a891419dee P 3ec57096ec2041b59bf3457ba5712e01 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:01:56.317253 27463 raft_consensus.cc:2454] T 0b3e8728954c430288a520a891419dee P 134b3b306d91453ebf1ed1314b21d3a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3df6aff1795d498bbfbd1046d78be724 in term 1.
I20260924 16:01:56.317425 27604 raft_consensus.cc:2454] T 0b3e8728954c430288a520a891419dee P 3ec57096ec2041b59bf3457ba5712e01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3df6aff1795d498bbfbd1046d78be724 in term 1.
I20260924 16:01:56.317572 27251 leader_election.cc:302] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 134b3b306d91453ebf1ed1314b21d3a8, 3df6aff1795d498bbfbd1046d78be724; no voters:
I20260924 16:01:56.317760 27683 raft_consensus.cc:2781] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:01:56.318029 27683 raft_consensus.cc:686] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724 [term 1 LEADER]: Becoming Leader. State: Replica: 3df6aff1795d498bbfbd1046d78be724, State: Running, Role: LEADER
I20260924 16:01:56.318318 27683 consensus_queue.cc:237] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } }
I20260924 16:01:56.319703 27719 tablet_replica.cc:439] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724: TxnStatusTablet state changed. Reason: New leader 3df6aff1795d498bbfbd1046d78be724. Latest consensus state: current_term: 1 leader_uuid: "3df6aff1795d498bbfbd1046d78be724" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } } }
I20260924 16:01:56.319819 27719 tablet_replica.cc:442] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:01:56.320073 27721 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 16:01:56.320273 27721 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 16:01:56.320895 27161 catalog_manager.cc:5495] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3df6aff1795d498bbfbd1046d78be724 (127.25.109.1). New cstate: current_term: 1 leader_uuid: "3df6aff1795d498bbfbd1046d78be724" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } health_report { overall_health: UNKNOWN } } }
I20260924 16:01:56.357167 27715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:56.352604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:40269 (local address 127.0.0.1:41960)
0924 16:01:56.352827 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:56.352847 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:56.352938 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:56.353308 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:56.353310 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:56.353316 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:01:56.353407 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:56.353412 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.354427 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.354429 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:56.355109 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:56.355118 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.355785 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.355791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:56.355878 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:56.355933 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:01:56.356822 (+ 889us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:01:56.356858 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 16:01:56.356948 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 16:01:56.357022 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 16:01:56.357187 27674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:56.352733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41960 (local address 127.25.109.1:40269)
0924 16:01:56.352885 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:56.352890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:56.352908 (+ 18us) server_negotiation.cc:407] Connection header received
0924 16:01:56.353092 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:56.353096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:56.353183 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:56.353294 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:01:56.353516 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.354319 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.355221 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.355609 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.355714 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:56.356103 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:01:56.356729 (+ 626us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:01:56.356843 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:56.356974 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 16:01:56.357033 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:01:56.360333 27604 raft_consensus.cc:1261] T 0b3e8728954c430288a520a891419dee P 3ec57096ec2041b59bf3457ba5712e01 [term 1 FOLLOWER]: Refusing update from remote peer 3df6aff1795d498bbfbd1046d78be724: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:01:56.360391 27463 raft_consensus.cc:1261] T 0b3e8728954c430288a520a891419dee P 134b3b306d91453ebf1ed1314b21d3a8 [term 1 FOLLOWER]: Refusing update from remote peer 3df6aff1795d498bbfbd1046d78be724: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:01:56.360991 27719 consensus_queue.cc:1035] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:01:56.361192 27683 consensus_queue.cc:1035] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724 [LEADER]: Connected to new peer: Peer: permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:01:56.363649 27682 tablet_replica.cc:439] T 0b3e8728954c430288a520a891419dee P 3ec57096ec2041b59bf3457ba5712e01: TxnStatusTablet state changed. Reason: New leader 3df6aff1795d498bbfbd1046d78be724. Latest consensus state: current_term: 1 leader_uuid: "3df6aff1795d498bbfbd1046d78be724" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } } }
I20260924 16:01:56.363762 27682 tablet_replica.cc:442] T 0b3e8728954c430288a520a891419dee P 3ec57096ec2041b59bf3457ba5712e01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:01:56.363885 27684 tablet_replica.cc:439] T 0b3e8728954c430288a520a891419dee P 134b3b306d91453ebf1ed1314b21d3a8: TxnStatusTablet state changed. Reason: New leader 3df6aff1795d498bbfbd1046d78be724. Latest consensus state: current_term: 1 leader_uuid: "3df6aff1795d498bbfbd1046d78be724" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } } }
I20260924 16:01:56.363993 27684 tablet_replica.cc:442] T 0b3e8728954c430288a520a891419dee P 134b3b306d91453ebf1ed1314b21d3a8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:01:56.364820 27719 tablet_replica.cc:439] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3df6aff1795d498bbfbd1046d78be724" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } } }
I20260924 16:01:56.364935 27719 tablet_replica.cc:442] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:01:56.365783 27682 tablet_replica.cc:439] T 0b3e8728954c430288a520a891419dee P 3ec57096ec2041b59bf3457ba5712e01: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3df6aff1795d498bbfbd1046d78be724" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } } }
I20260924 16:01:56.365891 27682 tablet_replica.cc:442] T 0b3e8728954c430288a520a891419dee P 3ec57096ec2041b59bf3457ba5712e01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:01:56.366439 27694 mvcc.cc:205] Tried to move back new op lower bound from 7332928374206320640 to 7332928374042898432. Current Snapshot: MvccSnapshot[applied={T|T < 7332928374206320640 or (T in {7332928374206320640})}]
I20260924 16:01:56.368664 27719 tablet_replica.cc:439] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3df6aff1795d498bbfbd1046d78be724" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } } }
I20260924 16:01:56.368804 27719 tablet_replica.cc:442] T 0b3e8728954c430288a520a891419dee P 3df6aff1795d498bbfbd1046d78be724: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:01:56.370056 27684 tablet_replica.cc:439] T 0b3e8728954c430288a520a891419dee P 134b3b306d91453ebf1ed1314b21d3a8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3df6aff1795d498bbfbd1046d78be724" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 } } peers { permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 } } peers { permanent_uuid: "3ec57096ec2041b59bf3457ba5712e01" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42955 } } }
I20260924 16:01:56.370189 27684 tablet_replica.cc:442] T 0b3e8728954c430288a520a891419dee P 134b3b306d91453ebf1ed1314b21d3a8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:01:56.376719 27688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:56.373172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39220 (local address 127.25.109.3:42955)
0924 16:01:56.373293 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:56.373299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:56.373313 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:01:56.373390 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:56.373394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:56.373482 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:56.373589 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:01:56.373846 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.374355 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.375211 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.375458 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.375573 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:56.375846 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:01:56.376291 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:01:56.376455 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:56.376532 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 16:01:56.376587 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 16:01:56.388067 27674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:56.384390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:45225 (local address 127.25.109.1:40269)
0924 16:01:56.384547 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:56.384550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:56.384564 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:01:56.384684 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:56.384686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:56.384731 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:56.384796 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:01:56.385069 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.385683 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.386971 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.387195 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.387282 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:56.387660 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:01:56.387728 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:01:56.387785 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:56.387915 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 16:01:56.387959 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 16:01:56.388137 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:56.384291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:40269 (local address 127.25.109.3:45225)
0924 16:01:56.384467 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:56.384484 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:56.384576 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:56.384829 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:56.384832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:56.384839 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:01:56.384957 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:56.384963 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.385794 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.385798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:56.386759 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:56.386771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.387316 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.387319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:56.387425 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:56.387479 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:01:56.387816 (+ 337us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:01:56.387824 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 16:01:56.387892 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 16:01:56.387986 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 16:01:56.395706 27463 raft_consensus.cc:1261] T 9d195fcd041f4540ac8db1f5fedff170 P 134b3b306d91453ebf1ed1314b21d3a8 [term 1 FOLLOWER]: Refusing update from remote peer 3ec57096ec2041b59bf3457ba5712e01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:01:56.395882 27321 raft_consensus.cc:1261] T 9d195fcd041f4540ac8db1f5fedff170 P 3df6aff1795d498bbfbd1046d78be724 [term 1 FOLLOWER]: Refusing update from remote peer 3ec57096ec2041b59bf3457ba5712e01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:01:56.396255 27682 consensus_queue.cc:1035] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "134b3b306d91453ebf1ed1314b21d3a8" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 43127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:01:56.396519 27708 consensus_queue.cc:1035] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3df6aff1795d498bbfbd1046d78be724" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 40269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:01:56.420536 27688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:56.416240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:34915 (local address 127.25.109.3:42955)
0924 16:01:56.416359 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:56.416364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:56.416379 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:01:56.416538 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:56.416542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:56.416591 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:56.416677 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:01:56.417074 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.417903 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.418771 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.419123 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.419227 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:56.419852 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:01:56.419950 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:01:56.420035 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:56.420248 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 16:01:56.420326 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 16:01:56.421103 27733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:56.415950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:42955 (local address 127.25.109.1:34915)
0924 16:01:56.416293 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:56.416311 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:56.416439 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:56.416811 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:56.416814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:56.416823 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:01:56.416948 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:56.416955 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.418016 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.418019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:56.418640 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:56.418648 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.419247 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.419251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:56.419357 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:56.419425 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:01:56.420117 (+ 692us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:01:56.420128 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 16:01:56.420774 (+ 646us) client_negotiation.cc:240] Negotiation successful
0924 16:01:56.420894 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":119,"threads_started":1}
I20260924 16:01:56.451174 27161 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:01:56.451354 27161 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:01:56.453891 27161 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 973b347f0b28409f991e362af9865bfe: Sending DeleteTablet for 3 replicas of tablet 9d195fcd041f4540ac8db1f5fedff170
I20260924 16:01:56.454509 27300 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d195fcd041f4540ac8db1f5fedff170 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:01:56 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:41950
I20260924 16:01:56.454733 27703 tablet_replica.cc:330] T 9d195fcd041f4540ac8db1f5fedff170 P 3df6aff1795d498bbfbd1046d78be724: stopping tablet replica
I20260924 16:01:56.454753 27442 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d195fcd041f4540ac8db1f5fedff170 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:01:56 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:39172
I20260924 16:01:56.454865 27703 raft_consensus.cc:2229] T 9d195fcd041f4540ac8db1f5fedff170 P 3df6aff1795d498bbfbd1046d78be724 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:01:56.454952 27701 tablet_replica.cc:330] T 9d195fcd041f4540ac8db1f5fedff170 P 134b3b306d91453ebf1ed1314b21d3a8: stopping tablet replica
I20260924 16:01:56.454988 27703 raft_consensus.cc:2258] T 9d195fcd041f4540ac8db1f5fedff170 P 3df6aff1795d498bbfbd1046d78be724 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:01:56.455096 27701 raft_consensus.cc:2229] T 9d195fcd041f4540ac8db1f5fedff170 P 134b3b306d91453ebf1ed1314b21d3a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:01:56.455220 27701 raft_consensus.cc:2258] T 9d195fcd041f4540ac8db1f5fedff170 P 134b3b306d91453ebf1ed1314b21d3a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:01:56.455240 27584 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d195fcd041f4540ac8db1f5fedff170 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:01:56 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:39218
I20260924 16:01:56.455400 27702 tablet_replica.cc:330] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01: stopping tablet replica
I20260924 16:01:56.455519 27702 raft_consensus.cc:2229] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:01:56.455860 27702 raft_consensus.cc:2258] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:01:56.456112 27701 ts_tablet_manager.cc:1905] T 9d195fcd041f4540ac8db1f5fedff170 P 134b3b306d91453ebf1ed1314b21d3a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:01:56.456115 27703 ts_tablet_manager.cc:1905] T 9d195fcd041f4540ac8db1f5fedff170 P 3df6aff1795d498bbfbd1046d78be724: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:01:56.456887 27702 ts_tablet_manager.cc:1905] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:01:56.459158 27703 ts_tablet_manager.cc:1918] T 9d195fcd041f4540ac8db1f5fedff170 P 3df6aff1795d498bbfbd1046d78be724: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:01:56.459160 27701 ts_tablet_manager.cc:1918] T 9d195fcd041f4540ac8db1f5fedff170 P 134b3b306d91453ebf1ed1314b21d3a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:01:56.459230 27703 log.cc:1196] T 9d195fcd041f4540ac8db1f5fedff170 P 3df6aff1795d498bbfbd1046d78be724: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-0/wal/wals/9d195fcd041f4540ac8db1f5fedff170
I20260924 16:01:56.459235 27701 log.cc:1196] T 9d195fcd041f4540ac8db1f5fedff170 P 134b3b306d91453ebf1ed1314b21d3a8: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-1/wal/wals/9d195fcd041f4540ac8db1f5fedff170
I20260924 16:01:56.459504 27703 ts_tablet_manager.cc:1939] T 9d195fcd041f4540ac8db1f5fedff170 P 3df6aff1795d498bbfbd1046d78be724: Deleting consensus metadata
I20260924 16:01:56.459510 27701 ts_tablet_manager.cc:1939] T 9d195fcd041f4540ac8db1f5fedff170 P 134b3b306d91453ebf1ed1314b21d3a8: Deleting consensus metadata
I20260924 16:01:56.459771 27702 ts_tablet_manager.cc:1918] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:01:56.459836 27702 log.cc:1196] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265708917108-26036-0/minicluster-data/ts-2/wal/wals/9d195fcd041f4540ac8db1f5fedff170
I20260924 16:01:56.460059 27702 ts_tablet_manager.cc:1939] T 9d195fcd041f4540ac8db1f5fedff170 P 3ec57096ec2041b59bf3457ba5712e01: Deleting consensus metadata
I20260924 16:01:56.460361 27146 catalog_manager.cc:4841] TS 134b3b306d91453ebf1ed1314b21d3a8 (127.25.109.2:43127): tablet 9d195fcd041f4540ac8db1f5fedff170 (table test-table [id=f6ba1cd4404844ec9ae5ef9eb4948dbf]) successfully deleted
I20260924 16:01:56.461119 27147 catalog_manager.cc:4841] TS 3ec57096ec2041b59bf3457ba5712e01 (127.25.109.3:42955): tablet 9d195fcd041f4540ac8db1f5fedff170 (table test-table [id=f6ba1cd4404844ec9ae5ef9eb4948dbf]) successfully deleted
I20260924 16:01:56.461822 27149 catalog_manager.cc:4841] TS 3df6aff1795d498bbfbd1046d78be724 (127.25.109.1:40269): tablet 9d195fcd041f4540ac8db1f5fedff170 (table test-table [id=f6ba1cd4404844ec9ae5ef9eb4948dbf]) successfully deleted
Sep 24 16:01:56 dist-test-slave-n106 krb5kdc[27117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265715, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.109.1@KRBTEST.COM
I20260924 16:01:56.475368 27674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:56.461477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41974 (local address 127.25.109.1:40269)
0924 16:01:56.461594 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:56.461611 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:56.462360 (+ 749us) server_negotiation.cc:407] Connection header received
0924 16:01:56.462550 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:56.462554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:56.462616 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:56.462671 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:56.463724 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.464512 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.465759 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.466100 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.466180 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:56.470837 (+ 4657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:56.470852 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:56.470871 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:56.470901 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:56.473047 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:56.473558 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:56.473564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:56.473566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:56.473692 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:56.473998 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:56.474002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:56.474004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:56.474289 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:56.474461 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:56.474891 (+ 430us) server_negotiation.cc:299] Negotiation successful
0924 16:01:56.475016 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 16:01:56.477257 27361 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41974
I20260924 16:01:56.494117 27212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:56.480800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48726 (local address 127.25.109.62:33761)
0924 16:01:56.480941 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:56.480946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:56.484730 (+ 3784us) server_negotiation.cc:407] Connection header received
0924 16:01:56.484923 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:56.484927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:56.484982 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:56.485034 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:56.486006 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.486852 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.488031 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:56.488373 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:56.488506 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:56.490430 (+ 1924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:56.490434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:56.490437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:56.490468 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:56.492232 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:56.492775 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:56.492779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:56.492781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:56.492873 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:56.493180 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:56.493183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:56.493185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:56.493400 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:56.493529 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:56.493795 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 16:01:56.493920 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260924 16:01:56.494603 27161 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48726
I20260924 16:01:56.496349 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 27235
I20260924 16:01:56.504848 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 27376
I20260924 16:01:56.514437 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 27519
I20260924 16:01:56.521425 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 27130
2026-09-24T16:01:56Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2703 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:01:56 dist-test-slave-n106 krb5kdc[27743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:01:56 dist-test-slave-n106 krb5kdc[27743](info): set up 2 sockets
Sep 24 16:01:56 dist-test-slave-n106 krb5kdc[27743](info): commencing operation
krb5kdc: starting...
W20260924 16:01:57.581312 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:01:57 dist-test-slave-n106 krb5kdc[27743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:01:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:01:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
I20260924 16:01:57.698799 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:36735
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:38783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:36735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:01:57.809294 27756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:01:57.809556 27756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:01:57.809597 27756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:01:57.813512 27756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:01:57.813581 27756 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 16:01:57.813599 27756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:01:57.813617 27756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:01:57.813633 27756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:01:57.813648 27756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:01:57.818784 27756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:38783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:36735
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:36735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:01:57.819782 27756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:01:57.820777 27756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:01:57.829236 27763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:01:57.829267 27765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:01:57.829490 27756 server_base.cc:1034] running on GCE node
W20260924 16:01:57.829651 27762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:01:57.830226 27756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:01:57.831298 27756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:01:57.832504 27756 hybrid_clock.cc:648] HybridClock initialized: now 1790265717832481 us; error 25 us; skew 500 ppm
Sep 24 16:01:57 dist-test-slave-n106 krb5kdc[27743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265717, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:01:57.835271 27756 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:01:57.836184 27756 webserver.cc:460] Webserver started at http://127.25.109.62:44971/ using document root <none> and password file <none>
I20260924 16:01:57.836799 27756 fs_manager.cc:359] Metadata directory not provided
I20260924 16:01:57.836861 27756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:01:57.837105 27756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:01:57.839099 27756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "0cdf45660a5941b28125b81a6b3d0a6e"
format_stamp: "Formatted at 2026-09-24 16:01:57 on dist-test-slave-n106"
server_key: "fee3cf7364999ed423c70790054bdf46"
server_key_iv: "fedadfb89431e7119005baa62aa339e8"
server_key_version: "encryptionkey@0"
I20260924 16:01:57.839669 27756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "0cdf45660a5941b28125b81a6b3d0a6e"
format_stamp: "Formatted at 2026-09-24 16:01:57 on dist-test-slave-n106"
server_key: "fee3cf7364999ed423c70790054bdf46"
server_key_iv: "fedadfb89431e7119005baa62aa339e8"
server_key_version: "encryptionkey@0"
I20260924 16:01:57.843427 27756 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 16:01:57.846017 27772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:57.847170 27756 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:01:57.847296 27756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "0cdf45660a5941b28125b81a6b3d0a6e"
format_stamp: "Formatted at 2026-09-24 16:01:57 on dist-test-slave-n106"
server_key: "fee3cf7364999ed423c70790054bdf46"
server_key_iv: "fedadfb89431e7119005baa62aa339e8"
server_key_version: "encryptionkey@0"
I20260924 16:01:57.847402 27756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:01:57.864826 27756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:01:57.876750 27756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:01:57.876947 27756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:01:57.884800 27756 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:36735
I20260924 16:01:57.884830 27836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:36735 every 8 connection(s)
I20260924 16:01:57.885730 27756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:01:57.888525 27837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:01:57.894722 27837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cdf45660a5941b28125b81a6b3d0a6e: Bootstrap starting.
I20260924 16:01:57.894886 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 27756
I20260924 16:01:57.895012 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:01:57.895309 26036 external_mini_cluster.cc:1428] Setting key d4c9e5594eb3b4fe09ed2dba2f61f56c
I20260924 16:01:57.897505 27837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cdf45660a5941b28125b81a6b3d0a6e: Neither blocks nor log segments found. Creating new log.
I20260924 16:01:57.898257 27837 log.cc:824] T 00000000000000000000000000000000 P 0cdf45660a5941b28125b81a6b3d0a6e: Log is configured to *not* fsync() on all Append() calls
I20260924 16:01:57.900113 27837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cdf45660a5941b28125b81a6b3d0a6e: No bootstrap required, opened a new log
Sep 24 16:01:57 dist-test-slave-n106 krb5kdc[27743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:01:57.903949 27837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cdf45660a5941b28125b81a6b3d0a6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cdf45660a5941b28125b81a6b3d0a6e" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 36735 } }
I20260924 16:01:57.904212 27837 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cdf45660a5941b28125b81a6b3d0a6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:01:57.904286 27837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cdf45660a5941b28125b81a6b3d0a6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cdf45660a5941b28125b81a6b3d0a6e, State: Initialized, Role: FOLLOWER
I20260924 16:01:57.904821 27837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cdf45660a5941b28125b81a6b3d0a6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cdf45660a5941b28125b81a6b3d0a6e" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 36735 } }
I20260924 16:01:57.904953 27837 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cdf45660a5941b28125b81a6b3d0a6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:01:57.905040 27837 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cdf45660a5941b28125b81a6b3d0a6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:01:57.905141 27837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cdf45660a5941b28125b81a6b3d0a6e [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:01:57.906352 27837 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cdf45660a5941b28125b81a6b3d0a6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cdf45660a5941b28125b81a6b3d0a6e" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 36735 } }
I20260924 16:01:57.906725 27837 leader_election.cc:302] T 00000000000000000000000000000000 P 0cdf45660a5941b28125b81a6b3d0a6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cdf45660a5941b28125b81a6b3d0a6e; no voters:
I20260924 16:01:57.907063 27837 leader_election.cc:288] T 00000000000000000000000000000000 P 0cdf45660a5941b28125b81a6b3d0a6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:01:57.907173 27842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cdf45660a5941b28125b81a6b3d0a6e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:01:57.907392 27842 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cdf45660a5941b28125b81a6b3d0a6e [term 1 LEADER]: Becoming Leader. State: Replica: 0cdf45660a5941b28125b81a6b3d0a6e, State: Running, Role: LEADER
I20260924 16:01:57.907724 27842 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cdf45660a5941b28125b81a6b3d0a6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cdf45660a5941b28125b81a6b3d0a6e" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 36735 } }
I20260924 16:01:57.908358 27837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cdf45660a5941b28125b81a6b3d0a6e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:01:57.909392 27843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cdf45660a5941b28125b81a6b3d0a6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cdf45660a5941b28125b81a6b3d0a6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cdf45660a5941b28125b81a6b3d0a6e" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 36735 } } }
I20260924 16:01:57.909518 27843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cdf45660a5941b28125b81a6b3d0a6e [sys.catalog]: This master's current role is: LEADER
I20260924 16:01:57.909843 27844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cdf45660a5941b28125b81a6b3d0a6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cdf45660a5941b28125b81a6b3d0a6e. Latest consensus state: current_term: 1 leader_uuid: "0cdf45660a5941b28125b81a6b3d0a6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cdf45660a5941b28125b81a6b3d0a6e" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 36735 } } }
I20260924 16:01:57.909930 27844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cdf45660a5941b28125b81a6b3d0a6e [sys.catalog]: This master's current role is: LEADER
I20260924 16:01:57.910769 27840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:57.896708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40848 (local address 127.25.109.62:36735)
0924 16:01:57.897164 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:57.897180 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:57.897216 (+ 36us) server_negotiation.cc:407] Connection header received
0924 16:01:57.897907 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:57.897927 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:57.898268 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:57.898640 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:57.899427 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:57.900538 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:57.901404 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:57.901883 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:57.902066 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:57.904781 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:57.904785 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:57.904797 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:57.904823 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:57.906883 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:57.907445 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:57.907453 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:57.907459 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:57.907594 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:57.908092 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:57.908095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:57.908097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:57.908634 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:57.908845 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:57.909355 (+ 510us) server_negotiation.cc:299] Negotiation successful
0924 16:01:57.909678 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":157,"threads_started":1}
W20260924 16:01:57.913415 27857 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0cdf45660a5941b28125b81a6b3d0a6e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:01:57.913513 27857 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:01:57.913587 27851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:01:57.914273 27851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:01:57.919831 27851 catalog_manager.cc:1347] Generated new cluster ID: 5eb2a61ff5f3499893605624d0f261f4
I20260924 16:01:57.919911 27851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:01:57.936144 27851 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:01:57.937088 27851 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:01:57.944391 27851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cdf45660a5941b28125b81a6b3d0a6e: Generated new TSK 0
I20260924 16:01:57.945186 27851 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
I20260924 16:01:57.979370 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:0
--local_ip_for_outbound_sockets=127.25.109.1
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:36735
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:38783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:01:58.089123 27863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:01:58.089362 27863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:01:58.089449 27863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:01:58.093519 27863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:01:58.093575 27863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:01:58.093648 27863 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 16:01:58.093690 27863 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:01:58.093775 27863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:01:58.098990 27863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:38783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.109.62:36735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:01:58.100032 27863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:01:58.101115 27863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:01:58.110808 27868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:01:58.110911 27869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:01:58.110881 27871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:01:58.111016 27863 server_base.cc:1034] running on GCE node
I20260924 16:01:58.111455 27863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:01:58.112076 27863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:01:58.113247 27863 hybrid_clock.cc:648] HybridClock initialized: now 1790265718113231 us; error 27 us; skew 500 ppm
Sep 24 16:01:58 dist-test-slave-n106 krb5kdc[27743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265718, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:01:58.115903 27863 init.cc:377] Logged in from keytab as kudu/127.25.109.1@KRBTEST.COM (short username kudu)
I20260924 16:01:58.116794 27863 webserver.cc:460] Webserver started at http://127.25.109.1:46271/ using document root <none> and password file <none>
I20260924 16:01:58.117344 27863 fs_manager.cc:359] Metadata directory not provided
I20260924 16:01:58.117400 27863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:01:58.117568 27863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:01:58.119572 27863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-0/data/instance:
uuid: "d63f57e80946448f877730a1dd865a7d"
format_stamp: "Formatted at 2026-09-24 16:01:58 on dist-test-slave-n106"
server_key: "b148eed8dc6712ea29169bfe2d14ea5d"
server_key_iv: "fb08ffced31fee13bfcc07c5112ae697"
server_key_version: "encryptionkey@0"
I20260924 16:01:58.120090 27863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance:
uuid: "d63f57e80946448f877730a1dd865a7d"
format_stamp: "Formatted at 2026-09-24 16:01:58 on dist-test-slave-n106"
server_key: "b148eed8dc6712ea29169bfe2d14ea5d"
server_key_iv: "fb08ffced31fee13bfcc07c5112ae697"
server_key_version: "encryptionkey@0"
I20260924 16:01:58.123834 27863 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 16:01:58.126397 27879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:58.127662 27863 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 16:01:58.127801 27863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "d63f57e80946448f877730a1dd865a7d"
format_stamp: "Formatted at 2026-09-24 16:01:58 on dist-test-slave-n106"
server_key: "b148eed8dc6712ea29169bfe2d14ea5d"
server_key_iv: "fb08ffced31fee13bfcc07c5112ae697"
server_key_version: "encryptionkey@0"
I20260924 16:01:58.127919 27863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:01:58.140842 27863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:01:58.143877 27863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:01:58.144069 27863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:01:58.145879 27863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:01:58.145968 27863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:58.146041 27863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:01:58.146107 27863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:01:58 dist-test-slave-n106 krb5kdc[27743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265718, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:01:58.157349 27863 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:37237
I20260924 16:01:58.157542 27996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:37237 every 8 connection(s)
I20260924 16:01:58.158376 27863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:01:58.160548 27840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:58.146613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:37081 (local address 127.25.109.62:36735)
0924 16:01:58.146737 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:58.146742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:58.148480 (+ 1738us) server_negotiation.cc:407] Connection header received
0924 16:01:58.149635 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:58.149639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:58.149704 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:58.149791 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:58.151310 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.152191 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.153468 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.153719 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.153832 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:58.156683 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:58.156687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:58.156689 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:58.156725 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:58.158344 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:58.158906 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:58.158910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:58.158912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:58.159016 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:58.159488 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:58.159491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:58.159492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:58.159684 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:58.159766 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:58.160234 (+ 468us) server_negotiation.cc:299] Negotiation successful
0924 16:01:58.160349 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 16:01:58.161355 27888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:58.147714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:36735 (local address 127.25.109.1:37081)
0924 16:01:58.148308 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:58.148349 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:58.149416 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:58.150083 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:58.150090 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:58.150527 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:01:58.151140 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:58.151156 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.152324 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.152327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:58.153337 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:58.153347 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.153881 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.153885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:58.154135 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:58.154811 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:01:58.154831 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:01:58.156521 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:01:58.158467 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:58.158473 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:58.158487 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:58.158791 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:58.159196 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:58.159200 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:58.159203 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:58.159346 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:58.159790 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:01:58.159795 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:01:58.160021 (+ 226us) client_negotiation.cc:769] Sending connection context
0924 16:01:58.160170 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 16:01:58.160465 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":150,"threads_started":1}
I20260924 16:01:58.162627 27997 heartbeater.cc:347] Connected to a master server at 127.25.109.62:36735
I20260924 16:01:58.162768 27997 heartbeater.cc:464] Registering TS with master...
I20260924 16:01:58.163321 27997 heartbeater.cc:511] Master 127.25.109.62:36735 requested a full tablet report, sending...
I20260924 16:01:58.165117 27789 ts_manager.cc:194] Registered new tserver with Master: d63f57e80946448f877730a1dd865a7d (127.25.109.1:37237)
I20260924 16:01:58.165566 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 27863
I20260924 16:01:58.165689 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance
I20260924 16:01:58.165961 26036 external_mini_cluster.cc:1428] Setting key 9b62c4f2f64d38c0033cb1d4073ec077
I20260924 16:01:58.166504 27789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.1@KRBTEST.COM'} at 127.25.109.1:37081
I20260924 16:01:58.175004 28003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:58.164976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:36735 (local address 127.25.109.1:41305)
0924 16:01:58.167591 (+ 2615us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:58.167615 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:58.167717 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:58.167972 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:58.167977 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:58.168215 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:01:58.168385 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:58.168396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.169359 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.169364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:58.170002 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:58.170011 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.170619 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.170621 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:58.170689 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:58.171219 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:01:58.171231 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:01:58.171544 (+ 313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:01:58.173567 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:58.173570 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:58.173571 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:58.173770 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:58.174068 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:58.174070 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:58.174072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:58.174147 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:58.174579 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:01:58.174581 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:01:58.174667 (+ 86us) client_negotiation.cc:769] Sending connection context
0924 16:01:58.174747 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 16:01:58.174853 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2523,"thread_start_us":87,"threads_started":1}
I20260924 16:01:58.175150 27840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:58.165055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:41305 (local address 127.25.109.62:36735)
0924 16:01:58.165166 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:58.165172 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:58.167699 (+ 2527us) server_negotiation.cc:407] Connection header received
0924 16:01:58.167800 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:58.167803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:58.167847 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:58.167927 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:58.168575 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.169217 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.170121 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.170481 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.170615 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:58.171687 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:58.171694 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:58.171700 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:58.171730 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:58.173441 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:58.173868 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:58.173874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:58.173879 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:58.173971 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:58.174236 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:58.174241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:58.174246 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:58.174474 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:58.174595 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:58.174836 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 16:01:58.174970 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
WARNING: no policy specified for kudu/127.25.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
I20260924 16:01:58.195173 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:0
--local_ip_for_outbound_sockets=127.25.109.2
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:36735
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:38783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:01:58.303694 28006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:01:58.303926 28006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:01:58.303972 28006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:01:58.307973 28006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:01:58.308024 28006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:01:58.308063 28006 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 16:01:58.308120 28006 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:01:58.308175 28006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:01:58.313383 28006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:38783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.109.62:36735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:01:58.314337 28006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:01:58.315320 28006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:01:58.325860 28014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:01:58.325881 28011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:01:58.326047 28006 server_base.cc:1034] running on GCE node
W20260924 16:01:58.325860 28012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:01:58.326538 28006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:01:58.327180 28006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:01:58.328372 28006 hybrid_clock.cc:648] HybridClock initialized: now 1790265718328354 us; error 26 us; skew 500 ppm
Sep 24 16:01:58 dist-test-slave-n106 krb5kdc[27743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265718, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:01:58.331162 28006 init.cc:377] Logged in from keytab as kudu/127.25.109.2@KRBTEST.COM (short username kudu)
I20260924 16:01:58.332064 28006 webserver.cc:460] Webserver started at http://127.25.109.2:37195/ using document root <none> and password file <none>
I20260924 16:01:58.332690 28006 fs_manager.cc:359] Metadata directory not provided
I20260924 16:01:58.332785 28006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:01:58.333016 28006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:01:58.335011 28006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-1/data/instance:
uuid: "a8f55ec33d6c418f97c5180d7e6c0e98"
format_stamp: "Formatted at 2026-09-24 16:01:58 on dist-test-slave-n106"
server_key: "835bb7f089ab1e3465bd7877f34c865d"
server_key_iv: "d0fe337ce9b887f571130d6c1e78b674"
server_key_version: "encryptionkey@0"
I20260924 16:01:58.335579 28006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance:
uuid: "a8f55ec33d6c418f97c5180d7e6c0e98"
format_stamp: "Formatted at 2026-09-24 16:01:58 on dist-test-slave-n106"
server_key: "835bb7f089ab1e3465bd7877f34c865d"
server_key_iv: "d0fe337ce9b887f571130d6c1e78b674"
server_key_version: "encryptionkey@0"
I20260924 16:01:58.339263 28006 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 16:01:58.341667 28021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:58.342861 28006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:01:58.342999 28006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "a8f55ec33d6c418f97c5180d7e6c0e98"
format_stamp: "Formatted at 2026-09-24 16:01:58 on dist-test-slave-n106"
server_key: "835bb7f089ab1e3465bd7877f34c865d"
server_key_iv: "d0fe337ce9b887f571130d6c1e78b674"
server_key_version: "encryptionkey@0"
I20260924 16:01:58.343111 28006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:01:58.371490 28006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:01:58.374966 28006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:01:58.375182 28006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:01:58.376775 28006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:01:58.376845 28006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:58.376919 28006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:01:58.376960 28006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:01:58 dist-test-slave-n106 krb5kdc[27743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265718, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:01:58.387842 28006 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:46761
I20260924 16:01:58.387907 28138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:46761 every 8 connection(s)
I20260924 16:01:58.389014 28006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:01:58.391108 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 28006
I20260924 16:01:58.391197 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance
I20260924 16:01:58.391391 27840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:58.377499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:42771 (local address 127.25.109.62:36735)
0924 16:01:58.377665 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:58.377672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:58.378978 (+ 1306us) server_negotiation.cc:407] Connection header received
0924 16:01:58.380556 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:58.380560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:58.380628 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:58.380739 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:58.382171 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.383083 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.384380 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.384759 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.384890 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:58.387430 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:58.387435 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:58.387440 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:58.387476 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:58.389221 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:58.389830 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:58.389834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:58.389835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:58.389926 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:58.390286 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:58.390290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:58.390292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:58.390482 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:58.390589 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:58.391096 (+ 507us) server_negotiation.cc:299] Negotiation successful
0924 16:01:58.391216 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 16:01:58.391436 26036 external_mini_cluster.cc:1428] Setting key a9719ddaa381341e4f97525dd966ac77
I20260924 16:01:58.392305 28031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:58.378364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:36735 (local address 127.25.109.2:42771)
0924 16:01:58.378803 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:58.378852 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:58.380317 (+ 1465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:58.380948 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:58.380957 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:58.381445 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:01:58.381999 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:58.382020 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.383241 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.383245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:58.384240 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:58.384251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.384916 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.384918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:58.385061 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:58.385669 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:01:58.385692 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:01:58.387257 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:01:58.389348 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:58.389352 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:58.389364 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:58.389725 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:58.390040 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:58.390045 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:58.390047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:58.390183 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:58.390599 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:01:58.390607 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:01:58.390840 (+ 233us) client_negotiation.cc:769] Sending connection context
0924 16:01:58.391046 (+ 206us) client_negotiation.cc:240] Negotiation successful
0924 16:01:58.391348 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":119,"threads_started":1}
I20260924 16:01:58.393543 28139 heartbeater.cc:347] Connected to a master server at 127.25.109.62:36735
I20260924 16:01:58.393697 28139 heartbeater.cc:464] Registering TS with master...
I20260924 16:01:58.394292 28139 heartbeater.cc:511] Master 127.25.109.62:36735 requested a full tablet report, sending...
I20260924 16:01:58.395413 27790 ts_manager.cc:194] Registered new tserver with Master: a8f55ec33d6c418f97c5180d7e6c0e98 (127.25.109.2:46761)
I20260924 16:01:58.396224 27790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.2@KRBTEST.COM'} at 127.25.109.2:42771
WARNING: no policy specified for kudu/127.25.109.3@KRBTEST.COM; defaulting to no policy
I20260924 16:01:58.403406 28146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:58.395685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:36735 (local address 127.25.109.2:44355)
0924 16:01:58.395988 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:58.396001 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:58.396093 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:58.396349 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:58.396352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:58.396608 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:01:58.396732 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:58.396739 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.397830 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.397834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:58.398447 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:58.398456 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.399117 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.399121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:58.399228 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:58.399763 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:01:58.399774 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:01:58.400064 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:01:58.401982 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:58.401985 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:58.401987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:58.402175 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:58.402460 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:58.402463 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:58.402485 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:58.402586 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:58.403021 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:01:58.403023 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:01:58.403094 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 16:01:58.403176 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 16:01:58.403272 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":89,"threads_started":1}
I20260924 16:01:58.403499 27840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:58.395769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:44355 (local address 127.25.109.62:36735)
0924 16:01:58.395871 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:58.395875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:58.396079 (+ 204us) server_negotiation.cc:407] Connection header received
0924 16:01:58.396164 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:58.396169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:58.396220 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:58.396315 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:58.396855 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.397688 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.398572 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.398994 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.399124 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:58.400213 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:58.400216 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:58.400220 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:58.400251 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:58.401873 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:58.402272 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:58.402276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:58.402278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:58.402368 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:58.402690 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:58.402695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:58.402700 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:58.402936 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:58.403019 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:58.403210 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 16:01:58.403294 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
I20260924 16:01:58.420233 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:0
--local_ip_for_outbound_sockets=127.25.109.3
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:36735
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:38783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:01:58.534703 28148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:01:58.534981 28148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:01:58.535056 28148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:01:58.539120 28148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:01:58.539202 28148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:01:58.539270 28148 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 16:01:58.539318 28148 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:01:58.539389 28148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:01:58.544591 28148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:38783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.109.62:36735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:01:58.545595 28148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:01:58.546653 28148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:01:58.556167 28153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:01:58.556222 28154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:01:58.556243 28148 server_base.cc:1034] running on GCE node
W20260924 16:01:58.556167 28156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:01:58.556952 28148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:01:58.557570 28148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:01:58.558753 28148 hybrid_clock.cc:648] HybridClock initialized: now 1790265718558733 us; error 30 us; skew 500 ppm
Sep 24 16:01:58 dist-test-slave-n106 krb5kdc[27743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265718, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:01:58.561429 28148 init.cc:377] Logged in from keytab as kudu/127.25.109.3@KRBTEST.COM (short username kudu)
I20260924 16:01:58.562258 28148 webserver.cc:460] Webserver started at http://127.25.109.3:45827/ using document root <none> and password file <none>
I20260924 16:01:58.562824 28148 fs_manager.cc:359] Metadata directory not provided
I20260924 16:01:58.562883 28148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:01:58.563047 28148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:01:58.565095 28148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-2/data/instance:
uuid: "94fcab817cd443e1b3b35d5e4d009699"
format_stamp: "Formatted at 2026-09-24 16:01:58 on dist-test-slave-n106"
server_key: "289cc12da5524d195515e804f2d49140"
server_key_iv: "b08f86727d6f4cd550780e52979371ec"
server_key_version: "encryptionkey@0"
I20260924 16:01:58.565603 28148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance:
uuid: "94fcab817cd443e1b3b35d5e4d009699"
format_stamp: "Formatted at 2026-09-24 16:01:58 on dist-test-slave-n106"
server_key: "289cc12da5524d195515e804f2d49140"
server_key_iv: "b08f86727d6f4cd550780e52979371ec"
server_key_version: "encryptionkey@0"
I20260924 16:01:58.569175 28148 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 16:01:58.571488 28163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:58.572723 28148 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 16:01:58.572817 28148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "94fcab817cd443e1b3b35d5e4d009699"
format_stamp: "Formatted at 2026-09-24 16:01:58 on dist-test-slave-n106"
server_key: "289cc12da5524d195515e804f2d49140"
server_key_iv: "b08f86727d6f4cd550780e52979371ec"
server_key_version: "encryptionkey@0"
I20260924 16:01:58.572897 28148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:01:58.598169 28148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:01:58.601114 28148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:01:58.601272 28148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:01:58.602741 28148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:01:58.602792 28148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:01:58.602836 28148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:01:58.602890 28148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:01:58 dist-test-slave-n106 krb5kdc[27743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265718, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:01:58.613463 28148 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:46755
I20260924 16:01:58.613726 28280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:46755 every 8 connection(s)
I20260924 16:01:58.614634 28148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:01:58.617194 27840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:58.603509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:40503 (local address 127.25.109.62:36735)
0924 16:01:58.603687 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:58.603695 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:58.604532 (+ 837us) server_negotiation.cc:407] Connection header received
0924 16:01:58.605494 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:58.605498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:58.605558 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:58.605654 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:58.607089 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.607926 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.609781 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.610216 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.610401 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:58.613518 (+ 3117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:58.613522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:58.613527 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:58.613556 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:58.615203 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:58.615846 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:58.615849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:58.615851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:58.615932 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:58.616235 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:58.616238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:58.616239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:58.616453 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:58.616535 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:58.616973 (+ 438us) server_negotiation.cc:299] Negotiation successful
0924 16:01:58.617071 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:01:58.617420 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 28148
I20260924 16:01:58.617543 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance
I20260924 16:01:58.617848 26036 external_mini_cluster.cc:1428] Setting key 02b6eb078f7867337f3fc22ed8febb6a
I20260924 16:01:58.618160 28172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:58.603898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:36735 (local address 127.25.109.3:40503)
0924 16:01:58.604350 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:58.604398 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:58.605282 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:58.605975 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:58.606001 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:58.606435 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:01:58.606908 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:58.606922 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.608052 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.608055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:58.609644 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:58.609658 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.610463 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.610468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:58.610698 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:58.611352 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:01:58.611368 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:01:58.613319 (+ 1951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:01:58.615361 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:58.615368 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:58.615385 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:58.615731 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:58.616015 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:58.616018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:58.616019 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:58.616148 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:58.616544 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:01:58.616549 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:01:58.616759 (+ 210us) client_negotiation.cc:769] Sending connection context
0924 16:01:58.616903 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 16:01:58.617138 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":123,"threads_started":1}
I20260924 16:01:58.619474 28281 heartbeater.cc:347] Connected to a master server at 127.25.109.62:36735
I20260924 16:01:58.619601 28281 heartbeater.cc:464] Registering TS with master...
I20260924 16:01:58.620117 28281 heartbeater.cc:511] Master 127.25.109.62:36735 requested a full tablet report, sending...
I20260924 16:01:58.621223 27789 ts_manager.cc:194] Registered new tserver with Master: 94fcab817cd443e1b3b35d5e4d009699 (127.25.109.3:46755)
I20260924 16:01:58.621960 27789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.3@KRBTEST.COM'} at 127.25.109.3:40503
I20260924 16:01:58.628142 28287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:58.621681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:36735 (local address 127.25.109.3:42231)
0924 16:01:58.622014 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:58.622025 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:58.622164 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:58.622436 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:58.622438 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:58.622594 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:01:58.622694 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:58.622699 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.623685 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.623687 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:58.624224 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:58.624231 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.624735 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.624738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:58.624804 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:58.625221 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:01:58.625232 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:01:58.625511 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:01:58.626938 (+ 1427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:58.626941 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:58.626943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:58.627134 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:58.627380 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:58.627382 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:58.627384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:58.627458 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:58.627770 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:01:58.627772 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:01:58.627841 (+ 69us) client_negotiation.cc:769] Sending connection context
0924 16:01:58.627919 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 16:01:58.628019 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":95,"threads_started":1}
I20260924 16:01:58.628170 27840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:58.621761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:42231 (local address 127.25.109.62:36735)
0924 16:01:58.621882 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:58.621886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:58.622100 (+ 214us) server_negotiation.cc:407] Connection header received
0924 16:01:58.622274 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:58.622278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:58.622322 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:58.622413 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:58.622812 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.623559 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.624331 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.624634 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.624718 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:58.625633 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:58.625636 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:58.625638 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:58.625658 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:58.626840 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:58.627228 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:58.627231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:58.627232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:58.627302 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:58.627536 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:58.627538 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:58.627539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:58.627689 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:58.627763 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:58.627983 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 16:01:58.628068 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 16:01:58.631814 26036 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 16:01:58 dist-test-slave-n106 krb5kdc[27743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 16:01:58 dist-test-slave-n106 krb5kdc[27743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:01:58.656634 27840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:58.648791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40850 (local address 127.25.109.62:36735)
0924 16:01:58.648919 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:58.648922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:58.648935 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:01:58.648981 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:58.648983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:58.649021 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:58.649090 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:58.649752 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.650300 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.651230 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.651438 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.651527 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:58.653823 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:58.653827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:58.653829 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:58.653855 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:58.655089 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:58.655488 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:58.655491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:58.655492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:58.655574 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:58.655817 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:58.655819 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:58.655820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:58.656000 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:58.656105 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:58.656301 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 16:01:58.656397 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 16:01:58.659458 27789 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:01:58.662565 27789 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:01:58.677560 28298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:58.671860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:46761 (local address 127.0.0.1:48956)
0924 16:01:58.672594 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:58.672607 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:58.672675 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:58.673163 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:58.673169 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:58.673192 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:01:58.673463 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:58.673470 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.674452 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.674457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:58.675738 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:58.675749 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.677007 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.677011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:58.677125 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:58.677171 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 16:01:58.677275 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 16:01:58.677396 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":72,"thread_start_us":90,"threads_started":1}
I20260924 16:01:58.677613 28299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:58.671155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:37237 (local address 127.0.0.1:44344)
0924 16:01:58.672190 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:58.672239 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:58.672460 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:58.673163 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:58.673169 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:58.673197 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:01:58.673463 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:58.673470 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.674861 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.674865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:58.676233 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:58.676242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.677301 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.677304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:58.677371 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:58.677392 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 16:01:58.677443 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 16:01:58.677493 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"mutex_wait_us":207,"spinlock_wait_cycles":1280,"thread_start_us":91,"threads_started":1}
I20260924 16:01:58.677690 28301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:58.671574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44344 (local address 127.25.109.1:37237)
0924 16:01:58.671882 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:58.671890 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:58.672320 (+ 430us) server_negotiation.cc:407] Connection header received
0924 16:01:58.672657 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:58.672663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:58.672881 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:58.672972 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:01:58.673567 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.674720 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.676359 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.677190 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.677297 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:58.677419 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:58.677472 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 16:01:58.677556 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":85,"threads_started":1}
I20260924 16:01:58.678166 28303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:58.672291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48956 (local address 127.25.109.2:46761)
0924 16:01:58.672752 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:58.672762 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:58.672788 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:01:58.672853 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:58.672857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:58.673008 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:58.673160 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:01:58.673615 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.674333 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.675856 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.676619 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.676780 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:58.676942 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:58.677924 (+ 982us) server_negotiation.cc:299] Negotiation successful
0924 16:01:58.678017 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":145,"threads_started":1}
I20260924 16:01:58.680800 27930 tablet_service.cc:1467] Processing CreateTablet for tablet 942e9c7843e649b69b9c6f8ea7d2aa23 (DEFAULT_TABLE table=test-table [id=99f0095ec10343098532db2969994ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:01:58.681857 28302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:58.671108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:46755 (local address 127.0.0.1:47012)
0924 16:01:58.672191 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:58.672241 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:58.672463 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:58.673198 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:58.673201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:58.673218 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:01:58.673369 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:58.673375 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.674861 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.674865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:58.676230 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:58.676242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.681449 (+ 5207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.681452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:58.681534 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:58.681554 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 16:01:58.681653 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 16:01:58.681725 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"mutex_wait_us":123,"spinlock_wait_cycles":64384,"thread_start_us":85,"threads_started":1}
I20260924 16:01:58.681926 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:58.671231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47012 (local address 127.25.109.3:46755)
0924 16:01:58.671989 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:58.672002 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:58.672331 (+ 329us) server_negotiation.cc:407] Connection header received
0924 16:01:58.672672 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:58.672679 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:58.672883 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:58.673005 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:01:58.673509 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.674721 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.678159 (+ 3438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.678773 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.678901 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:58.679065 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:58.681690 (+ 2625us) server_negotiation.cc:299] Negotiation successful
0924 16:01:58.681787 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":101,"threads_started":1}
I20260924 16:01:58.681958 27930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 942e9c7843e649b69b9c6f8ea7d2aa23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:01:58.682472 28072 tablet_service.cc:1467] Processing CreateTablet for tablet 942e9c7843e649b69b9c6f8ea7d2aa23 (DEFAULT_TABLE table=test-table [id=99f0095ec10343098532db2969994ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:01:58.683487 28072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 942e9c7843e649b69b9c6f8ea7d2aa23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:01:58.684562 28213 tablet_service.cc:1467] Processing CreateTablet for tablet 942e9c7843e649b69b9c6f8ea7d2aa23 (DEFAULT_TABLE table=test-table [id=99f0095ec10343098532db2969994ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:01:58.685350 28213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 942e9c7843e649b69b9c6f8ea7d2aa23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:01:58.689455 28304 tablet_bootstrap.cc:492] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d: Bootstrap starting.
I20260924 16:01:58.691015 28305 tablet_bootstrap.cc:492] T 942e9c7843e649b69b9c6f8ea7d2aa23 P a8f55ec33d6c418f97c5180d7e6c0e98: Bootstrap starting.
I20260924 16:01:58.692305 28304 tablet_bootstrap.cc:654] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d: Neither blocks nor log segments found. Creating new log.
I20260924 16:01:58.692446 28306 tablet_bootstrap.cc:492] T 942e9c7843e649b69b9c6f8ea7d2aa23 P 94fcab817cd443e1b3b35d5e4d009699: Bootstrap starting.
I20260924 16:01:58.693531 28305 tablet_bootstrap.cc:654] T 942e9c7843e649b69b9c6f8ea7d2aa23 P a8f55ec33d6c418f97c5180d7e6c0e98: Neither blocks nor log segments found. Creating new log.
I20260924 16:01:58.694167 28304 log.cc:824] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d: Log is configured to *not* fsync() on all Append() calls
I20260924 16:01:58.694330 28305 log.cc:824] T 942e9c7843e649b69b9c6f8ea7d2aa23 P a8f55ec33d6c418f97c5180d7e6c0e98: Log is configured to *not* fsync() on all Append() calls
I20260924 16:01:58.694705 28306 tablet_bootstrap.cc:654] T 942e9c7843e649b69b9c6f8ea7d2aa23 P 94fcab817cd443e1b3b35d5e4d009699: Neither blocks nor log segments found. Creating new log.
I20260924 16:01:58.695515 28306 log.cc:824] T 942e9c7843e649b69b9c6f8ea7d2aa23 P 94fcab817cd443e1b3b35d5e4d009699: Log is configured to *not* fsync() on all Append() calls
I20260924 16:01:58.696328 28304 tablet_bootstrap.cc:492] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d: No bootstrap required, opened a new log
I20260924 16:01:58.696357 28305 tablet_bootstrap.cc:492] T 942e9c7843e649b69b9c6f8ea7d2aa23 P a8f55ec33d6c418f97c5180d7e6c0e98: No bootstrap required, opened a new log
I20260924 16:01:58.696583 28305 ts_tablet_manager.cc:1397] T 942e9c7843e649b69b9c6f8ea7d2aa23 P a8f55ec33d6c418f97c5180d7e6c0e98: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 16:01:58.696591 28304 ts_tablet_manager.cc:1397] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260924 16:01:58.697557 28306 tablet_bootstrap.cc:492] T 942e9c7843e649b69b9c6f8ea7d2aa23 P 94fcab817cd443e1b3b35d5e4d009699: No bootstrap required, opened a new log
I20260924 16:01:58.697741 28306 ts_tablet_manager.cc:1397] T 942e9c7843e649b69b9c6f8ea7d2aa23 P 94fcab817cd443e1b3b35d5e4d009699: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 16:01:58.699874 28305 raft_consensus.cc:348] T 942e9c7843e649b69b9c6f8ea7d2aa23 P a8f55ec33d6c418f97c5180d7e6c0e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } } peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } }
I20260924 16:01:58.700160 28305 raft_consensus.cc:374] T 942e9c7843e649b69b9c6f8ea7d2aa23 P a8f55ec33d6c418f97c5180d7e6c0e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:01:58.700219 28305 raft_consensus.cc:729] T 942e9c7843e649b69b9c6f8ea7d2aa23 P a8f55ec33d6c418f97c5180d7e6c0e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8f55ec33d6c418f97c5180d7e6c0e98, State: Initialized, Role: FOLLOWER
I20260924 16:01:58.700753 28305 consensus_queue.cc:260] T 942e9c7843e649b69b9c6f8ea7d2aa23 P a8f55ec33d6c418f97c5180d7e6c0e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } } peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } }
I20260924 16:01:58.700824 28304 raft_consensus.cc:348] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } } peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } }
I20260924 16:01:58.701085 28304 raft_consensus.cc:374] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:01:58.701156 28304 raft_consensus.cc:729] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d63f57e80946448f877730a1dd865a7d, State: Initialized, Role: FOLLOWER
I20260924 16:01:58.701577 28304 consensus_queue.cc:260] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } } peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } }
I20260924 16:01:58.701648 28139 heartbeater.cc:503] Master 127.25.109.62:36735 was elected leader, sending a full tablet report...
I20260924 16:01:58.701555 28306 raft_consensus.cc:348] T 942e9c7843e649b69b9c6f8ea7d2aa23 P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } } peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } }
I20260924 16:01:58.701778 28306 raft_consensus.cc:374] T 942e9c7843e649b69b9c6f8ea7d2aa23 P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:01:58.701834 28306 raft_consensus.cc:729] T 942e9c7843e649b69b9c6f8ea7d2aa23 P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94fcab817cd443e1b3b35d5e4d009699, State: Initialized, Role: FOLLOWER
I20260924 16:01:58.701872 28305 ts_tablet_manager.cc:1428] T 942e9c7843e649b69b9c6f8ea7d2aa23 P a8f55ec33d6c418f97c5180d7e6c0e98: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 16:01:58.702255 28306 consensus_queue.cc:260] T 942e9c7843e649b69b9c6f8ea7d2aa23 P 94fcab817cd443e1b3b35d5e4d009699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } } peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } }
I20260924 16:01:58.702488 28304 ts_tablet_manager.cc:1428] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 16:01:58.702818 27997 heartbeater.cc:503] Master 127.25.109.62:36735 was elected leader, sending a full tablet report...
I20260924 16:01:58.703168 28306 ts_tablet_manager.cc:1428] T 942e9c7843e649b69b9c6f8ea7d2aa23 P 94fcab817cd443e1b3b35d5e4d009699: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 16:01:58.703238 28281 heartbeater.cc:503] Master 127.25.109.62:36735 was elected leader, sending a full tablet report...
I20260924 16:01:58.847209 28311 raft_consensus.cc:482] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:01:58.847492 28311 raft_consensus.cc:504] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } } peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } }
I20260924 16:01:58.848884 28311 leader_election.cc:288] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94fcab817cd443e1b3b35d5e4d009699 (127.25.109.3:46755), a8f55ec33d6c418f97c5180d7e6c0e98 (127.25.109.2:46761)
I20260924 16:01:58.853066 28314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:58.849146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:46755 (local address 127.25.109.1:46353)
0924 16:01:58.849553 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:58.849565 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:58.849667 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:58.849945 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:58.849948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:58.849966 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:01:58.850118 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:58.850126 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.850954 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.850957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:58.851903 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:58.851913 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.852612 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.852618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:58.852723 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:58.852746 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 16:01:58.852828 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 16:01:58.852883 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":139,"threads_started":1}
I20260924 16:01:58.853082 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:58.849264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:46353 (local address 127.25.109.3:46755)
0924 16:01:58.849428 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:58.849435 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:58.849647 (+ 212us) server_negotiation.cc:407] Connection header received
0924 16:01:58.849765 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:58.849771 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:58.849835 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:58.849914 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:01:58.850273 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.850848 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.852026 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.852494 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.852588 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:58.852632 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:58.852857 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 16:01:58.852947 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 16:01:58.853193 28303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:58.849330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:47513 (local address 127.25.109.2:46761)
0924 16:01:58.849457 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:58.849463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:58.849817 (+ 354us) server_negotiation.cc:407] Connection header received
0924 16:01:58.849953 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:58.849955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:58.850023 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:58.850104 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:01:58.850428 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.851025 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.852130 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.852598 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.852716 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:58.852769 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:58.852988 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 16:01:58.853064 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 16:01:58.853221 28313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:58.849170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:46761 (local address 127.25.109.1:47513)
0924 16:01:58.849709 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:58.849725 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:58.849828 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:58.850162 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:58.850167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:58.850184 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:01:58.850318 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:58.850326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.851196 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.851199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:58.852010 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:58.852021 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.852727 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.852731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:58.852855 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:58.852880 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 16:01:58.852957 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 16:01:58.853023 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":120,"threads_started":1}
I20260924 16:01:58.853905 28093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942e9c7843e649b69b9c6f8ea7d2aa23" candidate_uuid: "d63f57e80946448f877730a1dd865a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" is_pre_election: true
I20260924 16:01:58.854218 28234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942e9c7843e649b69b9c6f8ea7d2aa23" candidate_uuid: "d63f57e80946448f877730a1dd865a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94fcab817cd443e1b3b35d5e4d009699" is_pre_election: true
I20260924 16:01:58.854271 28093 raft_consensus.cc:2454] T 942e9c7843e649b69b9c6f8ea7d2aa23 P a8f55ec33d6c418f97c5180d7e6c0e98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d63f57e80946448f877730a1dd865a7d in term 0.
I20260924 16:01:58.854504 28234 raft_consensus.cc:2454] T 942e9c7843e649b69b9c6f8ea7d2aa23 P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d63f57e80946448f877730a1dd865a7d in term 0.
I20260924 16:01:58.854792 27880 leader_election.cc:302] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8f55ec33d6c418f97c5180d7e6c0e98, d63f57e80946448f877730a1dd865a7d; no voters:
I20260924 16:01:58.855099 28311 raft_consensus.cc:2781] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:01:58.855186 28311 raft_consensus.cc:482] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:01:58.855218 28311 raft_consensus.cc:3037] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:01:58.856297 28311 raft_consensus.cc:504] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } } peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } }
I20260924 16:01:58.856726 28311 leader_election.cc:288] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d [CANDIDATE]: Term 1 election: Requested vote from peers 94fcab817cd443e1b3b35d5e4d009699 (127.25.109.3:46755), a8f55ec33d6c418f97c5180d7e6c0e98 (127.25.109.2:46761)
I20260924 16:01:58.857089 28234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942e9c7843e649b69b9c6f8ea7d2aa23" candidate_uuid: "d63f57e80946448f877730a1dd865a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94fcab817cd443e1b3b35d5e4d009699"
I20260924 16:01:58.857115 28093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942e9c7843e649b69b9c6f8ea7d2aa23" candidate_uuid: "d63f57e80946448f877730a1dd865a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98"
I20260924 16:01:58.857213 28234 raft_consensus.cc:3037] T 942e9c7843e649b69b9c6f8ea7d2aa23 P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:01:58.857244 28093 raft_consensus.cc:3037] T 942e9c7843e649b69b9c6f8ea7d2aa23 P a8f55ec33d6c418f97c5180d7e6c0e98 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:01:58.858431 28234 raft_consensus.cc:2454] T 942e9c7843e649b69b9c6f8ea7d2aa23 P 94fcab817cd443e1b3b35d5e4d009699 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d63f57e80946448f877730a1dd865a7d in term 1.
I20260924 16:01:58.858556 28093 raft_consensus.cc:2454] T 942e9c7843e649b69b9c6f8ea7d2aa23 P a8f55ec33d6c418f97c5180d7e6c0e98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d63f57e80946448f877730a1dd865a7d in term 1.
I20260924 16:01:58.858820 27881 leader_election.cc:302] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94fcab817cd443e1b3b35d5e4d009699, d63f57e80946448f877730a1dd865a7d; no voters:
I20260924 16:01:58.858996 28311 raft_consensus.cc:2781] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:01:58.859257 28311 raft_consensus.cc:686] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d [term 1 LEADER]: Becoming Leader. State: Replica: d63f57e80946448f877730a1dd865a7d, State: Running, Role: LEADER
I20260924 16:01:58.859589 28311 consensus_queue.cc:237] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } } peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } }
I20260924 16:01:58.862641 27789 catalog_manager.cc:5495] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d reported cstate change: term changed from 0 to 1, leader changed from <none> to d63f57e80946448f877730a1dd865a7d (127.25.109.1). New cstate: current_term: 1 leader_uuid: "d63f57e80946448f877730a1dd865a7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } health_report { overall_health: UNKNOWN } } }
W20260924 16:01:58.865973 28282 tablet.cc:2369] T 942e9c7843e649b69b9c6f8ea7d2aa23 P 94fcab817cd443e1b3b35d5e4d009699: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:01:58.890221 28140 tablet.cc:2369] T 942e9c7843e649b69b9c6f8ea7d2aa23 P a8f55ec33d6c418f97c5180d7e6c0e98: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:01:58.909682 27998 tablet.cc:2369] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 16:01:58 dist-test-slave-n106 krb5kdc[27743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265717, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:01:58.934445 27840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:58.926183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40852 (local address 127.25.109.62:36735)
0924 16:01:58.926340 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:58.926347 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:58.926566 (+ 219us) server_negotiation.cc:407] Connection header received
0924 16:01:58.926738 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:58.926741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:58.926785 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:58.926873 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:01:58.927558 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.928079 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.928923 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.929140 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.929249 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:58.931464 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:01:58.931467 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:01:58.931469 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:01:58.931489 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:01:58.932756 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:58.933209 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:58.933215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:58.933216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:58.933292 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:01:58.933567 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:01:58.933572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:01:58.933573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:01:58.933743 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:01:58.933848 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:58.934138 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 16:01:58.934290 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":45}
I20260924 16:01:58.934445 28322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:58.926126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:36735 (local address 127.0.0.1:40852)
0924 16:01:58.926465 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:58.926477 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:58.926628 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:58.926891 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:58.926894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:58.927333 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:01:58.927430 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:58.927437 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.928189 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.928195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:58.928800 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:58.928808 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.929253 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.929256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:58.929350 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:58.929886 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:01:58.929898 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:01:58.931319 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:01:58.932859 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:58.932871 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:58.932878 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:58.933110 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:58.933387 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:01:58.933390 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:01:58.933392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:01:58.933472 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:01:58.933849 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:01:58.933857 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:01:58.933970 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 16:01:58.934082 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 16:01:58.934239 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":125,"threads_started":1}
I20260924 16:01:58.938501 27789 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:40852:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 16:01:58.939286 27789 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:01:58.944356 27930 tablet_service.cc:1467] Processing CreateTablet for tablet ab86cbe30364493c8b4975c8a39dc31f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9ad0ab6145e349fea93607cbef73f62b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:01:58.944341 28072 tablet_service.cc:1467] Processing CreateTablet for tablet ab86cbe30364493c8b4975c8a39dc31f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9ad0ab6145e349fea93607cbef73f62b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:01:58.944530 28213 tablet_service.cc:1467] Processing CreateTablet for tablet ab86cbe30364493c8b4975c8a39dc31f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9ad0ab6145e349fea93607cbef73f62b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:01:58.945180 28072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab86cbe30364493c8b4975c8a39dc31f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:01:58.945180 27930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab86cbe30364493c8b4975c8a39dc31f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:01:58.945225 28213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab86cbe30364493c8b4975c8a39dc31f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:01:58.948159 28304 tablet_bootstrap.cc:492] T ab86cbe30364493c8b4975c8a39dc31f P d63f57e80946448f877730a1dd865a7d: Bootstrap starting.
I20260924 16:01:58.948513 28306 tablet_bootstrap.cc:492] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699: Bootstrap starting.
I20260924 16:01:58.948702 28305 tablet_bootstrap.cc:492] T ab86cbe30364493c8b4975c8a39dc31f P a8f55ec33d6c418f97c5180d7e6c0e98: Bootstrap starting.
I20260924 16:01:58.949501 28304 tablet_bootstrap.cc:654] T ab86cbe30364493c8b4975c8a39dc31f P d63f57e80946448f877730a1dd865a7d: Neither blocks nor log segments found. Creating new log.
I20260924 16:01:58.949524 28306 tablet_bootstrap.cc:654] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699: Neither blocks nor log segments found. Creating new log.
I20260924 16:01:58.949581 28305 tablet_bootstrap.cc:654] T ab86cbe30364493c8b4975c8a39dc31f P a8f55ec33d6c418f97c5180d7e6c0e98: Neither blocks nor log segments found. Creating new log.
I20260924 16:01:58.950779 28304 tablet_bootstrap.cc:492] T ab86cbe30364493c8b4975c8a39dc31f P d63f57e80946448f877730a1dd865a7d: No bootstrap required, opened a new log
I20260924 16:01:58.950873 28304 ts_tablet_manager.cc:1397] T ab86cbe30364493c8b4975c8a39dc31f P d63f57e80946448f877730a1dd865a7d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 16:01:58.951010 28306 tablet_bootstrap.cc:492] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699: No bootstrap required, opened a new log
I20260924 16:01:58.951117 28306 ts_tablet_manager.cc:1397] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 16:01:58.951385 28304 raft_consensus.cc:348] T ab86cbe30364493c8b4975c8a39dc31f P d63f57e80946448f877730a1dd865a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:01:58.951483 28304 raft_consensus.cc:374] T ab86cbe30364493c8b4975c8a39dc31f P d63f57e80946448f877730a1dd865a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:01:58.951505 28304 raft_consensus.cc:729] T ab86cbe30364493c8b4975c8a39dc31f P d63f57e80946448f877730a1dd865a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d63f57e80946448f877730a1dd865a7d, State: Initialized, Role: FOLLOWER
I20260924 16:01:58.951598 28306 raft_consensus.cc:348] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:01:58.951647 28304 consensus_queue.cc:260] T ab86cbe30364493c8b4975c8a39dc31f P d63f57e80946448f877730a1dd865a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:01:58.951716 28306 raft_consensus.cc:374] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:01:58.951756 28306 raft_consensus.cc:729] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94fcab817cd443e1b3b35d5e4d009699, State: Initialized, Role: FOLLOWER
I20260924 16:01:58.951928 28304 ts_tablet_manager.cc:1428] T ab86cbe30364493c8b4975c8a39dc31f P d63f57e80946448f877730a1dd865a7d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:01:58.951906 28306 consensus_queue.cc:260] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:01:58.952171 28306 ts_tablet_manager.cc:1428] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260924 16:01:58.952159 28311 tablet_replica.cc:439] T ab86cbe30364493c8b4975c8a39dc31f P d63f57e80946448f877730a1dd865a7d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } } }
I20260924 16:01:58.952283 28311 tablet_replica.cc:442] T ab86cbe30364493c8b4975c8a39dc31f P d63f57e80946448f877730a1dd865a7d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:01:58.952397 28312 tablet_replica.cc:439] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } } }
I20260924 16:01:58.952565 28312 tablet_replica.cc:442] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:01:58.953428 28305 tablet_bootstrap.cc:492] T ab86cbe30364493c8b4975c8a39dc31f P a8f55ec33d6c418f97c5180d7e6c0e98: No bootstrap required, opened a new log
I20260924 16:01:58.953516 28305 ts_tablet_manager.cc:1397] T ab86cbe30364493c8b4975c8a39dc31f P a8f55ec33d6c418f97c5180d7e6c0e98: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260924 16:01:58.954308 28305 raft_consensus.cc:348] T ab86cbe30364493c8b4975c8a39dc31f P a8f55ec33d6c418f97c5180d7e6c0e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:01:58.954427 28305 raft_consensus.cc:374] T ab86cbe30364493c8b4975c8a39dc31f P a8f55ec33d6c418f97c5180d7e6c0e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:01:58.954463 28305 raft_consensus.cc:729] T ab86cbe30364493c8b4975c8a39dc31f P a8f55ec33d6c418f97c5180d7e6c0e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8f55ec33d6c418f97c5180d7e6c0e98, State: Initialized, Role: FOLLOWER
I20260924 16:01:58.954591 28305 consensus_queue.cc:260] T ab86cbe30364493c8b4975c8a39dc31f P a8f55ec33d6c418f97c5180d7e6c0e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:01:58.954903 28305 ts_tablet_manager.cc:1428] T ab86cbe30364493c8b4975c8a39dc31f P a8f55ec33d6c418f97c5180d7e6c0e98: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 16:01:58.954869 28310 tablet_replica.cc:439] T ab86cbe30364493c8b4975c8a39dc31f P a8f55ec33d6c418f97c5180d7e6c0e98: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } } }
I20260924 16:01:58.954995 28310 tablet_replica.cc:442] T ab86cbe30364493c8b4975c8a39dc31f P a8f55ec33d6c418f97c5180d7e6c0e98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:01:58.970371 28312 raft_consensus.cc:482] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:01:58.970530 28312 raft_consensus.cc:504] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:01:58.971808 28312 leader_election.cc:288] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d63f57e80946448f877730a1dd865a7d (127.25.109.1:37237), a8f55ec33d6c418f97c5180d7e6c0e98 (127.25.109.2:46761)
I20260924 16:01:58.976200 28172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:58.972032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:46761 (local address 127.25.109.3:39233)
0924 16:01:58.972169 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:58.972187 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:58.972279 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:58.972590 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:58.972594 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:58.972629 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:01:58.972771 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:58.972782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.973709 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.973711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:58.975076 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:58.975088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.975839 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.975842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:58.975937 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:58.975958 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 16:01:58.976018 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 16:01:58.976077 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 16:01:58.976259 28303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:58.972142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:39233 (local address 127.25.109.2:46761)
0924 16:01:58.972259 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:58.972266 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:58.972285 (+ 19us) server_negotiation.cc:407] Connection header received
0924 16:01:58.972381 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:58.972384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:58.972459 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:58.972572 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:01:58.972886 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.973597 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.975239 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.975717 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.975831 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:58.975887 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:58.976057 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 16:01:58.976134 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 16:01:58.977015 28301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:58.972145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:32807 (local address 127.25.109.1:37237)
0924 16:01:58.972278 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:58.972282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:58.972729 (+ 447us) server_negotiation.cc:407] Connection header received
0924 16:01:58.972819 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:58.972821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:58.972889 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:58.972965 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:01:58.973292 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.973935 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.975702 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.976338 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.976473 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:58.976524 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:58.976814 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 16:01:58.976898 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 16:01:58.977056 28093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab86cbe30364493c8b4975c8a39dc31f" candidate_uuid: "94fcab817cd443e1b3b35d5e4d009699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" is_pre_election: true
I20260924 16:01:58.977151 28326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:58.972032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:37237 (local address 127.25.109.3:32807)
0924 16:01:58.972617 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:58.972629 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:58.972720 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:58.972985 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:58.972987 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:58.973002 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:01:58.973182 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:58.973187 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.974040 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.974044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:58.975586 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:58.975596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:58.976475 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:58.976479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:58.976643 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:58.976672 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 16:01:58.976791 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 16:01:58.976906 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":162,"thread_start_us":131,"threads_started":1}
I20260924 16:01:58.977200 28093 raft_consensus.cc:2454] T ab86cbe30364493c8b4975c8a39dc31f P a8f55ec33d6c418f97c5180d7e6c0e98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94fcab817cd443e1b3b35d5e4d009699 in term 0.
I20260924 16:01:58.977658 28164 leader_election.cc:302] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94fcab817cd443e1b3b35d5e4d009699, a8f55ec33d6c418f97c5180d7e6c0e98; no voters:
I20260924 16:01:58.977813 27951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab86cbe30364493c8b4975c8a39dc31f" candidate_uuid: "94fcab817cd443e1b3b35d5e4d009699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d63f57e80946448f877730a1dd865a7d" is_pre_election: true
I20260924 16:01:58.978008 28312 raft_consensus.cc:2781] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:01:58.978030 27951 raft_consensus.cc:2454] T ab86cbe30364493c8b4975c8a39dc31f P d63f57e80946448f877730a1dd865a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94fcab817cd443e1b3b35d5e4d009699 in term 0.
I20260924 16:01:58.978093 28312 raft_consensus.cc:482] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:01:58.978135 28312 raft_consensus.cc:3037] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:01:58.979182 28312 raft_consensus.cc:504] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:01:58.979482 28312 leader_election.cc:288] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699 [CANDIDATE]: Term 1 election: Requested vote from peers d63f57e80946448f877730a1dd865a7d (127.25.109.1:37237), a8f55ec33d6c418f97c5180d7e6c0e98 (127.25.109.2:46761)
I20260924 16:01:58.979844 27951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab86cbe30364493c8b4975c8a39dc31f" candidate_uuid: "94fcab817cd443e1b3b35d5e4d009699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d63f57e80946448f877730a1dd865a7d"
I20260924 16:01:58.979867 28093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab86cbe30364493c8b4975c8a39dc31f" candidate_uuid: "94fcab817cd443e1b3b35d5e4d009699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98"
I20260924 16:01:58.979966 27951 raft_consensus.cc:3037] T ab86cbe30364493c8b4975c8a39dc31f P d63f57e80946448f877730a1dd865a7d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:01:58.979978 28093 raft_consensus.cc:3037] T ab86cbe30364493c8b4975c8a39dc31f P a8f55ec33d6c418f97c5180d7e6c0e98 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:01:58.981194 28093 raft_consensus.cc:2454] T ab86cbe30364493c8b4975c8a39dc31f P a8f55ec33d6c418f97c5180d7e6c0e98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94fcab817cd443e1b3b35d5e4d009699 in term 1.
I20260924 16:01:58.981200 27951 raft_consensus.cc:2454] T ab86cbe30364493c8b4975c8a39dc31f P d63f57e80946448f877730a1dd865a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94fcab817cd443e1b3b35d5e4d009699 in term 1.
I20260924 16:01:58.981487 28166 leader_election.cc:302] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94fcab817cd443e1b3b35d5e4d009699, d63f57e80946448f877730a1dd865a7d; no voters:
I20260924 16:01:58.981657 28312 raft_consensus.cc:2781] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:01:58.981881 28312 raft_consensus.cc:686] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699 [term 1 LEADER]: Becoming Leader. State: Replica: 94fcab817cd443e1b3b35d5e4d009699, State: Running, Role: LEADER
I20260924 16:01:58.982235 28312 consensus_queue.cc:237] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:01:58.984625 28327 tablet_replica.cc:439] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699: TxnStatusTablet state changed. Reason: New leader 94fcab817cd443e1b3b35d5e4d009699. Latest consensus state: current_term: 1 leader_uuid: "94fcab817cd443e1b3b35d5e4d009699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } } }
I20260924 16:01:58.984745 28327 tablet_replica.cc:442] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:01:58.985170 28329 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 16:01:58.985404 28329 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 16:01:58.985879 27789 catalog_manager.cc:5495] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94fcab817cd443e1b3b35d5e4d009699 (127.25.109.3). New cstate: current_term: 1 leader_uuid: "94fcab817cd443e1b3b35d5e4d009699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } health_report { overall_health: HEALTHY } } }
I20260924 16:01:59.008828 28322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:59.004570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:46755 (local address 127.0.0.1:47016)
0924 16:01:59.004749 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:59.004766 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:59.004872 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:59.005304 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:59.005337 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:59.005347 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:01:59.005484 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:59.005493 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:59.006399 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:59.006402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:59.007017 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:59.007025 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:59.007510 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:59.007514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:59.007632 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:59.007680 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:01:59.008531 (+ 851us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:01:59.008547 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 16:01:59.008616 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 16:01:59.008706 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 16:01:59.009341 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:59.004695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47016 (local address 127.25.109.3:46755)
0924 16:01:59.004838 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:59.004843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:59.004860 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:01:59.005030 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:59.005035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:59.005140 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:59.005268 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:01:59.005624 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:59.006281 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:59.007162 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:59.007393 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:59.007506 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:59.007835 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:01:59.008392 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:01:59.008546 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:59.008629 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 16:01:59.008684 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 16:01:59.014678 28093 raft_consensus.cc:1261] T ab86cbe30364493c8b4975c8a39dc31f P a8f55ec33d6c418f97c5180d7e6c0e98 [term 1 FOLLOWER]: Refusing update from remote peer 94fcab817cd443e1b3b35d5e4d009699: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:01:59.014676 27951 raft_consensus.cc:1261] T ab86cbe30364493c8b4975c8a39dc31f P d63f57e80946448f877730a1dd865a7d [term 1 FOLLOWER]: Refusing update from remote peer 94fcab817cd443e1b3b35d5e4d009699: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:01:59.015337 28312 consensus_queue.cc:1035] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:01:59.015786 28312 consensus_queue.cc:1035] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:01:59.018906 28311 tablet_replica.cc:439] T ab86cbe30364493c8b4975c8a39dc31f P d63f57e80946448f877730a1dd865a7d: TxnStatusTablet state changed. Reason: New leader 94fcab817cd443e1b3b35d5e4d009699. Latest consensus state: current_term: 1 leader_uuid: "94fcab817cd443e1b3b35d5e4d009699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } } }
I20260924 16:01:59.019042 28311 tablet_replica.cc:442] T ab86cbe30364493c8b4975c8a39dc31f P d63f57e80946448f877730a1dd865a7d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:01:59.020010 28310 tablet_replica.cc:439] T ab86cbe30364493c8b4975c8a39dc31f P a8f55ec33d6c418f97c5180d7e6c0e98: TxnStatusTablet state changed. Reason: New leader 94fcab817cd443e1b3b35d5e4d009699. Latest consensus state: current_term: 1 leader_uuid: "94fcab817cd443e1b3b35d5e4d009699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } } }
I20260924 16:01:59.020136 28310 tablet_replica.cc:442] T ab86cbe30364493c8b4975c8a39dc31f P a8f55ec33d6c418f97c5180d7e6c0e98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:01:59.022399 28312 tablet_replica.cc:439] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "94fcab817cd443e1b3b35d5e4d009699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } } }
I20260924 16:01:59.022509 28312 tablet_replica.cc:442] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:01:59.022607 28332 mvcc.cc:205] Tried to move back new op lower bound from 7332928385076678656 to 7332928384953757696. Current Snapshot: MvccSnapshot[applied={T|T < 7332928385076678656}]
I20260924 16:01:59.022734 28311 tablet_replica.cc:439] T ab86cbe30364493c8b4975c8a39dc31f P d63f57e80946448f877730a1dd865a7d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "94fcab817cd443e1b3b35d5e4d009699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } } }
I20260924 16:01:59.022846 28311 tablet_replica.cc:442] T ab86cbe30364493c8b4975c8a39dc31f P d63f57e80946448f877730a1dd865a7d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:01:59.023454 28312 tablet_replica.cc:439] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "94fcab817cd443e1b3b35d5e4d009699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } } }
I20260924 16:01:59.023564 28312 tablet_replica.cc:442] T ab86cbe30364493c8b4975c8a39dc31f P 94fcab817cd443e1b3b35d5e4d009699: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:01:59.023816 28334 mvcc.cc:205] Tried to move back new op lower bound from 7332928385076678656 to 7332928384953757696. Current Snapshot: MvccSnapshot[applied={T|T < 7332928385076678656}]
I20260924 16:01:59.024583 28310 tablet_replica.cc:439] T ab86cbe30364493c8b4975c8a39dc31f P a8f55ec33d6c418f97c5180d7e6c0e98: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "94fcab817cd443e1b3b35d5e4d009699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } } }
I20260924 16:01:59.024726 28310 tablet_replica.cc:442] T ab86cbe30364493c8b4975c8a39dc31f P a8f55ec33d6c418f97c5180d7e6c0e98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:01:59.033983 28301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:59.029736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44356 (local address 127.25.109.1:37237)
0924 16:01:59.029846 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:59.029849 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:59.029977 (+ 128us) server_negotiation.cc:407] Connection header received
0924 16:01:59.030099 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:59.030103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:59.030164 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:59.030267 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:01:59.030487 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:59.031336 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:59.032328 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:59.032697 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:59.032801 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:59.033041 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:01:59.033561 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:01:59.033676 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:59.033761 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 16:01:59.033824 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 16:01:59.045343 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:59.041847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:52025 (local address 127.25.109.3:46755)
0924 16:01:59.042095 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:59.042100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:59.042163 (+ 63us) server_negotiation.cc:407] Connection header received
0924 16:01:59.042330 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:59.042334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:59.042389 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:59.042474 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:01:59.042732 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:59.043240 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:59.044146 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:59.044365 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:59.044525 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:59.044861 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:01:59.044931 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:01:59.045012 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:59.045175 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 16:01:59.045235 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260924 16:01:59.045753 28343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:59.041729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:46755 (local address 127.25.109.1:52025)
0924 16:01:59.042051 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:59.042066 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:59.042224 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:59.042484 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:59.042487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:59.042494 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:01:59.042625 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:59.042631 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:59.043350 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:59.043355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:59.044002 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:59.044011 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:59.044539 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:59.044543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:59.044640 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:59.044683 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:01:59.045018 (+ 335us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:01:59.045028 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 16:01:59.045516 (+ 488us) client_negotiation.cc:240] Negotiation successful
0924 16:01:59.045597 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":126,"threads_started":1}
I20260924 16:01:59.053136 28093 raft_consensus.cc:1261] T 942e9c7843e649b69b9c6f8ea7d2aa23 P a8f55ec33d6c418f97c5180d7e6c0e98 [term 1 FOLLOWER]: Refusing update from remote peer d63f57e80946448f877730a1dd865a7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:01:59.053442 28234 raft_consensus.cc:1261] T 942e9c7843e649b69b9c6f8ea7d2aa23 P 94fcab817cd443e1b3b35d5e4d009699 [term 1 FOLLOWER]: Refusing update from remote peer d63f57e80946448f877730a1dd865a7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:01:59.053591 28311 consensus_queue.cc:1035] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:01:59.054035 28315 consensus_queue.cc:1035] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:01:59.057926 28333 mvcc.cc:205] Tried to move back new op lower bound from 7332928385237045248 to 7332928384451371008. Current Snapshot: MvccSnapshot[applied={T|T < 7332928385237045248}]
I20260924 16:01:59.077517 28287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:59.073540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:37237 (local address 127.25.109.3:34479)
0924 16:01:59.073780 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:59.073793 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:59.073869 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:59.074093 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:59.074095 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:59.074105 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:01:59.074201 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:59.074205 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:59.075211 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:59.075214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:59.076205 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:59.076217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:59.076735 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:59.076739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:59.076841 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:59.076923 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:01:59.077241 (+ 318us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:01:59.077250 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 16:01:59.077316 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 16:01:59.077392 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260924 16:01:59.077517 28301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:59.073639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:34479 (local address 127.25.109.1:37237)
0924 16:01:59.073757 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:59.073762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:59.073832 (+ 70us) server_negotiation.cc:407] Connection header received
0924 16:01:59.073952 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:59.073961 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:59.074009 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:59.074091 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:01:59.074292 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:59.075073 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:59.076318 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:59.076609 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:59.076736 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:59.077062 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:01:59.077153 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:01:59.077237 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:59.077326 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 16:01:59.077380 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 16:01:59.103322 27788 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:01:59.103542 27788 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:01:59.106325 27788 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0cdf45660a5941b28125b81a6b3d0a6e: Sending DeleteTablet for 3 replicas of tablet 942e9c7843e649b69b9c6f8ea7d2aa23
I20260924 16:01:59.107014 28213 tablet_service.cc:1514] Processing DeleteTablet for tablet 942e9c7843e649b69b9c6f8ea7d2aa23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:01:59 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:47012
I20260924 16:01:59.107156 28072 tablet_service.cc:1514] Processing DeleteTablet for tablet 942e9c7843e649b69b9c6f8ea7d2aa23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:01:59 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:48956
I20260924 16:01:59.107621 28352 tablet_replica.cc:330] T 942e9c7843e649b69b9c6f8ea7d2aa23 P 94fcab817cd443e1b3b35d5e4d009699: stopping tablet replica
I20260924 16:01:59.107728 27930 tablet_service.cc:1514] Processing DeleteTablet for tablet 942e9c7843e649b69b9c6f8ea7d2aa23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:01:59 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:44344
I20260924 16:01:59.108032 28352 raft_consensus.cc:2229] T 942e9c7843e649b69b9c6f8ea7d2aa23 P 94fcab817cd443e1b3b35d5e4d009699 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:01:59.108224 28352 raft_consensus.cc:2258] T 942e9c7843e649b69b9c6f8ea7d2aa23 P 94fcab817cd443e1b3b35d5e4d009699 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:01:59.108889 27788 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:01:59.109334 27788 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:01:59.110244 28352 ts_tablet_manager.cc:1905] T 942e9c7843e649b69b9c6f8ea7d2aa23 P 94fcab817cd443e1b3b35d5e4d009699: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:01:59.113785 28353 tablet_replica.cc:330] T 942e9c7843e649b69b9c6f8ea7d2aa23 P a8f55ec33d6c418f97c5180d7e6c0e98: stopping tablet replica
I20260924 16:01:59.113788 28352 ts_tablet_manager.cc:1918] T 942e9c7843e649b69b9c6f8ea7d2aa23 P 94fcab817cd443e1b3b35d5e4d009699: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:01:59.113992 28352 log.cc:1196] T 942e9c7843e649b69b9c6f8ea7d2aa23 P 94fcab817cd443e1b3b35d5e4d009699: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-2/wal/wals/942e9c7843e649b69b9c6f8ea7d2aa23
I20260924 16:01:59.114144 28353 raft_consensus.cc:2229] T 942e9c7843e649b69b9c6f8ea7d2aa23 P a8f55ec33d6c418f97c5180d7e6c0e98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:01:59.114378 28353 raft_consensus.cc:2258] T 942e9c7843e649b69b9c6f8ea7d2aa23 P a8f55ec33d6c418f97c5180d7e6c0e98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:01:59.114353 28072 tablet_service.cc:1467] Processing CreateTablet for tablet 1f6aac62b0b740609491d6e9351cb6bc (DEFAULT_TABLE table=test-table [id=384f5358c9d345feb02a35eed7b43225]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:01:59.114432 28213 tablet_service.cc:1467] Processing CreateTablet for tablet 1f6aac62b0b740609491d6e9351cb6bc (DEFAULT_TABLE table=test-table [id=384f5358c9d345feb02a35eed7b43225]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:01:59.114620 28352 ts_tablet_manager.cc:1939] T 942e9c7843e649b69b9c6f8ea7d2aa23 P 94fcab817cd443e1b3b35d5e4d009699: Deleting consensus metadata
I20260924 16:01:59.114661 28072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f6aac62b0b740609491d6e9351cb6bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:01:59.114665 28213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f6aac62b0b740609491d6e9351cb6bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:01:59.115207 27930 tablet_service.cc:1467] Processing CreateTablet for tablet 1f6aac62b0b740609491d6e9351cb6bc (DEFAULT_TABLE table=test-table [id=384f5358c9d345feb02a35eed7b43225]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:01:59.115438 27930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f6aac62b0b740609491d6e9351cb6bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:01:59.117123 28306 tablet_bootstrap.cc:492] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699: Bootstrap starting.
I20260924 16:01:59.117271 28305 tablet_bootstrap.cc:492] T 1f6aac62b0b740609491d6e9351cb6bc P a8f55ec33d6c418f97c5180d7e6c0e98: Bootstrap starting.
I20260924 16:01:59.118110 28306 tablet_bootstrap.cc:654] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699: Neither blocks nor log segments found. Creating new log.
I20260924 16:01:59.118242 28305 tablet_bootstrap.cc:654] T 1f6aac62b0b740609491d6e9351cb6bc P a8f55ec33d6c418f97c5180d7e6c0e98: Neither blocks nor log segments found. Creating new log.
I20260924 16:01:59.118602 28304 tablet_bootstrap.cc:492] T 1f6aac62b0b740609491d6e9351cb6bc P d63f57e80946448f877730a1dd865a7d: Bootstrap starting.
I20260924 16:01:59.119575 28304 tablet_bootstrap.cc:654] T 1f6aac62b0b740609491d6e9351cb6bc P d63f57e80946448f877730a1dd865a7d: Neither blocks nor log segments found. Creating new log.
I20260924 16:01:59.120605 27774 catalog_manager.cc:4841] TS 94fcab817cd443e1b3b35d5e4d009699 (127.25.109.3:46755): tablet 942e9c7843e649b69b9c6f8ea7d2aa23 (table test-table [id=99f0095ec10343098532db2969994ae1]) successfully deleted
I20260924 16:01:59.121807 28353 ts_tablet_manager.cc:1905] T 942e9c7843e649b69b9c6f8ea7d2aa23 P a8f55ec33d6c418f97c5180d7e6c0e98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:01:59.122288 28304 tablet_bootstrap.cc:492] T 1f6aac62b0b740609491d6e9351cb6bc P d63f57e80946448f877730a1dd865a7d: No bootstrap required, opened a new log
I20260924 16:01:59.122378 28304 ts_tablet_manager.cc:1397] T 1f6aac62b0b740609491d6e9351cb6bc P d63f57e80946448f877730a1dd865a7d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 16:01:59.122804 28304 raft_consensus.cc:348] T 1f6aac62b0b740609491d6e9351cb6bc P d63f57e80946448f877730a1dd865a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:01:59.122961 28304 raft_consensus.cc:374] T 1f6aac62b0b740609491d6e9351cb6bc P d63f57e80946448f877730a1dd865a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:01:59.122998 28304 raft_consensus.cc:729] T 1f6aac62b0b740609491d6e9351cb6bc P d63f57e80946448f877730a1dd865a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d63f57e80946448f877730a1dd865a7d, State: Initialized, Role: FOLLOWER
I20260924 16:01:59.123109 28304 consensus_queue.cc:260] T 1f6aac62b0b740609491d6e9351cb6bc P d63f57e80946448f877730a1dd865a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:01:59.123354 28304 ts_tablet_manager.cc:1428] T 1f6aac62b0b740609491d6e9351cb6bc P d63f57e80946448f877730a1dd865a7d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:01:59.126230 28354 tablet_replica.cc:330] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d: stopping tablet replica
I20260924 16:01:59.126523 28353 ts_tablet_manager.cc:1918] T 942e9c7843e649b69b9c6f8ea7d2aa23 P a8f55ec33d6c418f97c5180d7e6c0e98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:01:59.126537 28354 raft_consensus.cc:2229] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:01:59.126679 28353 log.cc:1196] T 942e9c7843e649b69b9c6f8ea7d2aa23 P a8f55ec33d6c418f97c5180d7e6c0e98: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-1/wal/wals/942e9c7843e649b69b9c6f8ea7d2aa23
I20260924 16:01:59.126857 28354 raft_consensus.cc:2258] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:01:59.127127 28353 ts_tablet_manager.cc:1939] T 942e9c7843e649b69b9c6f8ea7d2aa23 P a8f55ec33d6c418f97c5180d7e6c0e98: Deleting consensus metadata
I20260924 16:01:59.128472 27773 catalog_manager.cc:4841] TS a8f55ec33d6c418f97c5180d7e6c0e98 (127.25.109.2:46761): tablet 942e9c7843e649b69b9c6f8ea7d2aa23 (table test-table [id=99f0095ec10343098532db2969994ae1]) successfully deleted
I20260924 16:01:59.128916 28354 ts_tablet_manager.cc:1905] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:01:59.129519 28306 tablet_bootstrap.cc:492] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699: No bootstrap required, opened a new log
I20260924 16:01:59.129612 28306 ts_tablet_manager.cc:1397] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20260924 16:01:59.130100 28306 raft_consensus.cc:348] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:01:59.130226 28306 raft_consensus.cc:374] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:01:59.130261 28306 raft_consensus.cc:729] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94fcab817cd443e1b3b35d5e4d009699, State: Initialized, Role: FOLLOWER
I20260924 16:01:59.130393 28306 consensus_queue.cc:260] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:01:59.130662 28306 ts_tablet_manager.cc:1428] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:01:59.131119 28305 tablet_bootstrap.cc:492] T 1f6aac62b0b740609491d6e9351cb6bc P a8f55ec33d6c418f97c5180d7e6c0e98: No bootstrap required, opened a new log
I20260924 16:01:59.131210 28305 ts_tablet_manager.cc:1397] T 1f6aac62b0b740609491d6e9351cb6bc P a8f55ec33d6c418f97c5180d7e6c0e98: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20260924 16:01:59.131690 28305 raft_consensus.cc:348] T 1f6aac62b0b740609491d6e9351cb6bc P a8f55ec33d6c418f97c5180d7e6c0e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:01:59.131819 28305 raft_consensus.cc:374] T 1f6aac62b0b740609491d6e9351cb6bc P a8f55ec33d6c418f97c5180d7e6c0e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:01:59.131851 28305 raft_consensus.cc:729] T 1f6aac62b0b740609491d6e9351cb6bc P a8f55ec33d6c418f97c5180d7e6c0e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8f55ec33d6c418f97c5180d7e6c0e98, State: Initialized, Role: FOLLOWER
I20260924 16:01:59.131970 28305 consensus_queue.cc:260] T 1f6aac62b0b740609491d6e9351cb6bc P a8f55ec33d6c418f97c5180d7e6c0e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:01:59.132236 28305 ts_tablet_manager.cc:1428] T 1f6aac62b0b740609491d6e9351cb6bc P a8f55ec33d6c418f97c5180d7e6c0e98: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:01:59.132795 28354 ts_tablet_manager.cc:1918] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:01:59.132895 28354 log.cc:1196] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-0/wal/wals/942e9c7843e649b69b9c6f8ea7d2aa23
I20260924 16:01:59.133280 28354 ts_tablet_manager.cc:1939] T 942e9c7843e649b69b9c6f8ea7d2aa23 P d63f57e80946448f877730a1dd865a7d: Deleting consensus metadata
I20260924 16:01:59.134581 27776 catalog_manager.cc:4841] TS d63f57e80946448f877730a1dd865a7d (127.25.109.1:37237): tablet 942e9c7843e649b69b9c6f8ea7d2aa23 (table test-table [id=99f0095ec10343098532db2969994ae1]) successfully deleted
I20260924 16:01:59.278007 28327 raft_consensus.cc:482] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:01:59.278141 28327 raft_consensus.cc:504] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:01:59.278542 28327 leader_election.cc:288] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d63f57e80946448f877730a1dd865a7d (127.25.109.1:37237), a8f55ec33d6c418f97c5180d7e6c0e98 (127.25.109.2:46761)
I20260924 16:01:59.278947 27951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f6aac62b0b740609491d6e9351cb6bc" candidate_uuid: "94fcab817cd443e1b3b35d5e4d009699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d63f57e80946448f877730a1dd865a7d" is_pre_election: true
I20260924 16:01:59.279104 27951 raft_consensus.cc:2454] T 1f6aac62b0b740609491d6e9351cb6bc P d63f57e80946448f877730a1dd865a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94fcab817cd443e1b3b35d5e4d009699 in term 0.
I20260924 16:01:59.279049 28093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f6aac62b0b740609491d6e9351cb6bc" candidate_uuid: "94fcab817cd443e1b3b35d5e4d009699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" is_pre_election: true
I20260924 16:01:59.279263 28093 raft_consensus.cc:2454] T 1f6aac62b0b740609491d6e9351cb6bc P a8f55ec33d6c418f97c5180d7e6c0e98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94fcab817cd443e1b3b35d5e4d009699 in term 0.
I20260924 16:01:59.279474 28166 leader_election.cc:302] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94fcab817cd443e1b3b35d5e4d009699, d63f57e80946448f877730a1dd865a7d; no voters:
I20260924 16:01:59.279668 28327 raft_consensus.cc:2781] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:01:59.279729 28327 raft_consensus.cc:482] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:01:59.279772 28327 raft_consensus.cc:3037] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:01:59.280895 28327 raft_consensus.cc:504] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:01:59.281296 28327 leader_election.cc:288] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699 [CANDIDATE]: Term 1 election: Requested vote from peers d63f57e80946448f877730a1dd865a7d (127.25.109.1:37237), a8f55ec33d6c418f97c5180d7e6c0e98 (127.25.109.2:46761)
I20260924 16:01:59.281594 27951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f6aac62b0b740609491d6e9351cb6bc" candidate_uuid: "94fcab817cd443e1b3b35d5e4d009699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d63f57e80946448f877730a1dd865a7d"
I20260924 16:01:59.281706 27951 raft_consensus.cc:3037] T 1f6aac62b0b740609491d6e9351cb6bc P d63f57e80946448f877730a1dd865a7d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:01:59.282575 27951 raft_consensus.cc:2454] T 1f6aac62b0b740609491d6e9351cb6bc P d63f57e80946448f877730a1dd865a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94fcab817cd443e1b3b35d5e4d009699 in term 1.
I20260924 16:01:59.282843 28093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f6aac62b0b740609491d6e9351cb6bc" candidate_uuid: "94fcab817cd443e1b3b35d5e4d009699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98"
I20260924 16:01:59.282866 28166 leader_election.cc:302] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94fcab817cd443e1b3b35d5e4d009699, d63f57e80946448f877730a1dd865a7d; no voters:
I20260924 16:01:59.282949 28093 raft_consensus.cc:3037] T 1f6aac62b0b740609491d6e9351cb6bc P a8f55ec33d6c418f97c5180d7e6c0e98 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:01:59.283052 28327 raft_consensus.cc:2781] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:01:59.283129 28327 raft_consensus.cc:686] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699 [term 1 LEADER]: Becoming Leader. State: Replica: 94fcab817cd443e1b3b35d5e4d009699, State: Running, Role: LEADER
I20260924 16:01:59.283259 28327 consensus_queue.cc:237] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:01:59.283988 28093 raft_consensus.cc:2454] T 1f6aac62b0b740609491d6e9351cb6bc P a8f55ec33d6c418f97c5180d7e6c0e98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94fcab817cd443e1b3b35d5e4d009699 in term 1.
I20260924 16:01:59.284910 27788 catalog_manager.cc:5495] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94fcab817cd443e1b3b35d5e4d009699 (127.25.109.3). New cstate: current_term: 1 leader_uuid: "94fcab817cd443e1b3b35d5e4d009699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } health_report { overall_health: HEALTHY } } }
I20260924 16:01:59.325677 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:59.322328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47032 (local address 127.25.109.3:46755)
0924 16:01:59.322460 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:59.322467 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:59.322483 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:01:59.322530 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:59.322534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:59.322605 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:59.322679 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:01:59.322965 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:59.323746 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:59.324740 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:59.324949 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:59.325038 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:59.325271 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:01:59.325343 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:01:59.325410 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:59.325481 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 16:01:59.325569 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 16:01:59.334031 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:59.331173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:45299 (local address 127.25.109.3:46755)
0924 16:01:59.331281 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 16:01:59.331284 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:01:59.331295 (+ 11us) server_negotiation.cc:407] Connection header received
0924 16:01:59.331419 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:01:59.331422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:01:59.331470 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:01:59.331530 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:01:59.331711 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:59.332395 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:59.333208 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:01:59.333407 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:01:59.333501 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:59.333699 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:01:59.333757 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:01:59.333809 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 16:01:59.333884 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 16:01:59.333921 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":49}
I20260924 16:01:59.334033 28287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:01:59.331114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:46755 (local address 127.25.109.3:45299)
0924 16:01:59.331244 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 16:01:59.331255 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 16:01:59.331316 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:01:59.331549 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:01:59.331551 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:01:59.331557 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:01:59.331627 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:59.331631 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:59.332563 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:59.332566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:59.333113 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:01:59.333120 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:01:59.333498 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:01:59.333500 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:01:59.333582 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:01:59.333605 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:01:59.333827 (+ 222us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:01:59.333831 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 16:01:59.333877 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 16:01:59.333934 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":15}
I20260924 16:01:59.339226 27951 raft_consensus.cc:1261] T 1f6aac62b0b740609491d6e9351cb6bc P d63f57e80946448f877730a1dd865a7d [term 1 FOLLOWER]: Refusing update from remote peer 94fcab817cd443e1b3b35d5e4d009699: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:01:59.339459 28093 raft_consensus.cc:1261] T 1f6aac62b0b740609491d6e9351cb6bc P a8f55ec33d6c418f97c5180d7e6c0e98 [term 1 FOLLOWER]: Refusing update from remote peer 94fcab817cd443e1b3b35d5e4d009699: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:01:59.339702 28360 consensus_queue.cc:1035] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:01:59.340025 28360 consensus_queue.cc:1035] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 16:01:59.352972 28332 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 16:02:00.907294 27788 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:02:00.907446 27788 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:02:00.910018 27788 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0cdf45660a5941b28125b81a6b3d0a6e: Sending DeleteTablet for 3 replicas of tablet 1f6aac62b0b740609491d6e9351cb6bc
I20260924 16:02:00.910557 28072 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f6aac62b0b740609491d6e9351cb6bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:00 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:48956
I20260924 16:02:00.910657 27930 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f6aac62b0b740609491d6e9351cb6bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:00 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:44344
I20260924 16:02:00.910831 28213 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f6aac62b0b740609491d6e9351cb6bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:00 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:47012
I20260924 16:02:00.911131 28386 tablet_replica.cc:330] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699: stopping tablet replica
I20260924 16:02:00.911150 28384 tablet_replica.cc:330] T 1f6aac62b0b740609491d6e9351cb6bc P a8f55ec33d6c418f97c5180d7e6c0e98: stopping tablet replica
I20260924 16:02:00.911270 28386 raft_consensus.cc:2229] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:02:00.911270 28384 raft_consensus.cc:2229] T 1f6aac62b0b740609491d6e9351cb6bc P a8f55ec33d6c418f97c5180d7e6c0e98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:00.911382 28384 raft_consensus.cc:2258] T 1f6aac62b0b740609491d6e9351cb6bc P a8f55ec33d6c418f97c5180d7e6c0e98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:00.911600 28385 tablet_replica.cc:330] T 1f6aac62b0b740609491d6e9351cb6bc P d63f57e80946448f877730a1dd865a7d: stopping tablet replica
I20260924 16:02:00.911610 28386 raft_consensus.cc:2258] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:00.911742 28385 raft_consensus.cc:2229] T 1f6aac62b0b740609491d6e9351cb6bc P d63f57e80946448f877730a1dd865a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:00.911854 28385 raft_consensus.cc:2258] T 1f6aac62b0b740609491d6e9351cb6bc P d63f57e80946448f877730a1dd865a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:00.912237 28386 ts_tablet_manager.cc:1905] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:00.912490 28384 ts_tablet_manager.cc:1905] T 1f6aac62b0b740609491d6e9351cb6bc P a8f55ec33d6c418f97c5180d7e6c0e98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:00.914951 28384 ts_tablet_manager.cc:1918] T 1f6aac62b0b740609491d6e9351cb6bc P a8f55ec33d6c418f97c5180d7e6c0e98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 16:02:00.915048 28384 log.cc:1196] T 1f6aac62b0b740609491d6e9351cb6bc P a8f55ec33d6c418f97c5180d7e6c0e98: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-1/wal/wals/1f6aac62b0b740609491d6e9351cb6bc
I20260924 16:02:00.915042 28386 ts_tablet_manager.cc:1918] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 16:02:00.915104 28386 log.cc:1196] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-2/wal/wals/1f6aac62b0b740609491d6e9351cb6bc
I20260924 16:02:00.915337 28384 ts_tablet_manager.cc:1939] T 1f6aac62b0b740609491d6e9351cb6bc P a8f55ec33d6c418f97c5180d7e6c0e98: Deleting consensus metadata
I20260924 16:02:00.915402 28386 ts_tablet_manager.cc:1939] T 1f6aac62b0b740609491d6e9351cb6bc P 94fcab817cd443e1b3b35d5e4d009699: Deleting consensus metadata
I20260924 16:02:00.916040 27773 catalog_manager.cc:4841] TS a8f55ec33d6c418f97c5180d7e6c0e98 (127.25.109.2:46761): tablet 1f6aac62b0b740609491d6e9351cb6bc (table test-table [id=384f5358c9d345feb02a35eed7b43225]) successfully deleted
I20260924 16:02:00.916153 27774 catalog_manager.cc:4841] TS 94fcab817cd443e1b3b35d5e4d009699 (127.25.109.3:46755): tablet 1f6aac62b0b740609491d6e9351cb6bc (table test-table [id=384f5358c9d345feb02a35eed7b43225]) successfully deleted
I20260924 16:02:00.916478 28385 ts_tablet_manager.cc:1905] T 1f6aac62b0b740609491d6e9351cb6bc P d63f57e80946448f877730a1dd865a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:00.919221 28385 ts_tablet_manager.cc:1918] T 1f6aac62b0b740609491d6e9351cb6bc P d63f57e80946448f877730a1dd865a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 16:02:00.919302 28385 log.cc:1196] T 1f6aac62b0b740609491d6e9351cb6bc P d63f57e80946448f877730a1dd865a7d: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-0/wal/wals/1f6aac62b0b740609491d6e9351cb6bc
I20260924 16:02:00.919621 28385 ts_tablet_manager.cc:1939] T 1f6aac62b0b740609491d6e9351cb6bc P d63f57e80946448f877730a1dd865a7d: Deleting consensus metadata
I20260924 16:02:00.920311 27776 catalog_manager.cc:4841] TS d63f57e80946448f877730a1dd865a7d (127.25.109.1:37237): tablet 1f6aac62b0b740609491d6e9351cb6bc (table test-table [id=384f5358c9d345feb02a35eed7b43225]) successfully deleted
Sep 24 16:02:00 dist-test-slave-n106 krb5kdc[27743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265720, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 16:02:00 dist-test-slave-n106 krb5kdc[27743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265720, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:02:00.938941 28394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:00.930364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40858 (local address 127.25.109.62:36735)
0924 16:02:00.930690 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:00.930698 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:00.930714 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:02:00.930835 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:00.930838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:00.930885 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:00.930962 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:00.931682 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:00.932275 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:00.933182 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:00.933409 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:00.933489 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:00.935912 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:00.935915 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:00.935917 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:00.935943 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:00.937523 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:00.937888 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:00.937891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:00.937892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:00.937968 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:00.938195 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:00.938197 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:00.938198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:00.938372 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:00.938474 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:00.938711 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 16:02:00.938811 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":102,"threads_started":1}
I20260924 16:02:00.941588 27788 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:02:00.942065 27788 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:02:00.946992 27930 tablet_service.cc:1467] Processing CreateTablet for tablet d4b72eb4ae214cb5ab2ca7aba9b3e854 (DEFAULT_TABLE table=test-table [id=d43c55f435764c68bcbd0414f8fbb983]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:00.947028 28072 tablet_service.cc:1467] Processing CreateTablet for tablet d4b72eb4ae214cb5ab2ca7aba9b3e854 (DEFAULT_TABLE table=test-table [id=d43c55f435764c68bcbd0414f8fbb983]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:00.947337 27930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b72eb4ae214cb5ab2ca7aba9b3e854. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:00.947337 28072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b72eb4ae214cb5ab2ca7aba9b3e854. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:00.949957 28395 tablet_bootstrap.cc:492] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P d63f57e80946448f877730a1dd865a7d: Bootstrap starting.
I20260924 16:02:00.950127 28213 tablet_service.cc:1467] Processing CreateTablet for tablet d4b72eb4ae214cb5ab2ca7aba9b3e854 (DEFAULT_TABLE table=test-table [id=d43c55f435764c68bcbd0414f8fbb983]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:00.950417 28213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b72eb4ae214cb5ab2ca7aba9b3e854. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:00.950927 28395 tablet_bootstrap.cc:654] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P d63f57e80946448f877730a1dd865a7d: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:00.951702 28396 tablet_bootstrap.cc:492] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98: Bootstrap starting.
I20260924 16:02:00.952669 28396 tablet_bootstrap.cc:654] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:00.953065 28398 tablet_bootstrap.cc:492] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P 94fcab817cd443e1b3b35d5e4d009699: Bootstrap starting.
I20260924 16:02:00.953871 28395 tablet_bootstrap.cc:492] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P d63f57e80946448f877730a1dd865a7d: No bootstrap required, opened a new log
I20260924 16:02:00.953956 28398 tablet_bootstrap.cc:654] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P 94fcab817cd443e1b3b35d5e4d009699: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:00.953989 28395 ts_tablet_manager.cc:1397] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P d63f57e80946448f877730a1dd865a7d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 16:02:00.954353 28395 raft_consensus.cc:348] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P d63f57e80946448f877730a1dd865a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:02:00.954479 28395 raft_consensus.cc:374] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P d63f57e80946448f877730a1dd865a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:00.954515 28395 raft_consensus.cc:729] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P d63f57e80946448f877730a1dd865a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d63f57e80946448f877730a1dd865a7d, State: Initialized, Role: FOLLOWER
I20260924 16:02:00.954695 28395 consensus_queue.cc:260] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P d63f57e80946448f877730a1dd865a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:02:00.955109 28395 ts_tablet_manager.cc:1428] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P d63f57e80946448f877730a1dd865a7d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:02:00.955194 28398 tablet_bootstrap.cc:492] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P 94fcab817cd443e1b3b35d5e4d009699: No bootstrap required, opened a new log
I20260924 16:02:00.955260 28396 tablet_bootstrap.cc:492] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98: No bootstrap required, opened a new log
I20260924 16:02:00.955289 28398 ts_tablet_manager.cc:1397] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P 94fcab817cd443e1b3b35d5e4d009699: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 16:02:00.955346 28396 ts_tablet_manager.cc:1397] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 16:02:00.955765 28398 raft_consensus.cc:348] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:02:00.955878 28398 raft_consensus.cc:374] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:00.955912 28398 raft_consensus.cc:729] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94fcab817cd443e1b3b35d5e4d009699, State: Initialized, Role: FOLLOWER
I20260924 16:02:00.955935 28396 raft_consensus.cc:348] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:02:00.956050 28396 raft_consensus.cc:374] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:00.956084 28396 raft_consensus.cc:729] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8f55ec33d6c418f97c5180d7e6c0e98, State: Initialized, Role: FOLLOWER
I20260924 16:02:00.956035 28398 consensus_queue.cc:260] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P 94fcab817cd443e1b3b35d5e4d009699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:02:00.956228 28396 consensus_queue.cc:260] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:02:00.956313 28398 ts_tablet_manager.cc:1428] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P 94fcab817cd443e1b3b35d5e4d009699: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:02:00.956617 28396 ts_tablet_manager.cc:1428] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:02:01.019658 28402 raft_consensus.cc:482] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:02:01.019886 28402 raft_consensus.cc:504] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:02:01.021163 28402 leader_election.cc:288] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d63f57e80946448f877730a1dd865a7d (127.25.109.1:37237), 94fcab817cd443e1b3b35d5e4d009699 (127.25.109.3:46755)
I20260924 16:02:01.025563 28403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:01.021353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:37237 (local address 127.25.109.2:34473)
0924 16:02:01.021736 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:01.021753 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:01.021869 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:01.022163 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:01.022166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:01.022182 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:01.022328 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:01.022334 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:01.023119 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:01.023123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:01.024085 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:01.024127 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:01.025037 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:01.025041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:01.025137 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:01.025193 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 16:02:01.025287 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 16:02:01.025382 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":24,"thread_start_us":108,"threads_started":1}
I20260924 16:02:01.025550 28405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:01.021490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:34473 (local address 127.25.109.1:37237)
0924 16:02:01.021888 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:01.021897 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:01.021917 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:02:01.021977 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:01.021980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:01.022032 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:01.022128 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:01.022453 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:01.022990 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:01.024250 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:01.024905 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:01.025041 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:01.025102 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:01.025306 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 16:02:01.025389 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":102,"threads_started":1}
I20260924 16:02:01.025559 28406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:01.021558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:33063 (local address 127.25.109.3:46755)
0924 16:02:01.021839 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:01.021852 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:01.021943 (+ 91us) server_negotiation.cc:407] Connection header received
0924 16:02:01.022116 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:01.022119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:01.022170 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:01.022264 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:01.022590 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:01.023088 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:01.024363 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:01.024932 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:01.025070 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:01.025137 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:01.025330 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 16:02:01.025420 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":121,"threads_started":1}
I20260924 16:02:01.025580 28404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:01.021456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:46755 (local address 127.25.109.2:33063)
0924 16:02:01.021837 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:01.021852 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:01.022000 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:01.022276 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:01.022279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:01.022296 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:01.022440 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:01.022445 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:01.023200 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:01.023205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:01.024233 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:01.024243 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:01.025070 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:01.025074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:01.025194 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:01.025222 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 16:02:01.025310 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 16:02:01.025412 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":45,"thread_start_us":122,"threads_started":1}
I20260924 16:02:01.026314 27951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b72eb4ae214cb5ab2ca7aba9b3e854" candidate_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d63f57e80946448f877730a1dd865a7d" is_pre_election: true
I20260924 16:02:01.026328 28234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b72eb4ae214cb5ab2ca7aba9b3e854" candidate_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94fcab817cd443e1b3b35d5e4d009699" is_pre_election: true
I20260924 16:02:01.026484 27951 raft_consensus.cc:2454] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P d63f57e80946448f877730a1dd865a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8f55ec33d6c418f97c5180d7e6c0e98 in term 0.
I20260924 16:02:01.026494 28234 raft_consensus.cc:2454] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8f55ec33d6c418f97c5180d7e6c0e98 in term 0.
I20260924 16:02:01.026889 28024 leader_election.cc:302] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8f55ec33d6c418f97c5180d7e6c0e98, d63f57e80946448f877730a1dd865a7d; no voters:
I20260924 16:02:01.027187 28402 raft_consensus.cc:2781] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:02:01.027251 28402 raft_consensus.cc:482] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:02:01.027290 28402 raft_consensus.cc:3037] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:01.028302 28402 raft_consensus.cc:504] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:02:01.028724 28402 leader_election.cc:288] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98 [CANDIDATE]: Term 1 election: Requested vote from peers d63f57e80946448f877730a1dd865a7d (127.25.109.1:37237), 94fcab817cd443e1b3b35d5e4d009699 (127.25.109.3:46755)
I20260924 16:02:01.029058 27951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b72eb4ae214cb5ab2ca7aba9b3e854" candidate_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d63f57e80946448f877730a1dd865a7d"
I20260924 16:02:01.029114 28234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b72eb4ae214cb5ab2ca7aba9b3e854" candidate_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94fcab817cd443e1b3b35d5e4d009699"
I20260924 16:02:01.029199 27951 raft_consensus.cc:3037] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P d63f57e80946448f877730a1dd865a7d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:01.029227 28234 raft_consensus.cc:3037] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:01.030369 28234 raft_consensus.cc:2454] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P 94fcab817cd443e1b3b35d5e4d009699 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8f55ec33d6c418f97c5180d7e6c0e98 in term 1.
I20260924 16:02:01.030369 27951 raft_consensus.cc:2454] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P d63f57e80946448f877730a1dd865a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8f55ec33d6c418f97c5180d7e6c0e98 in term 1.
I20260924 16:02:01.030668 28023 leader_election.cc:302] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94fcab817cd443e1b3b35d5e4d009699, a8f55ec33d6c418f97c5180d7e6c0e98; no voters:
I20260924 16:02:01.030843 28402 raft_consensus.cc:2781] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:01.031076 28402 raft_consensus.cc:686] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98 [term 1 LEADER]: Becoming Leader. State: Replica: a8f55ec33d6c418f97c5180d7e6c0e98, State: Running, Role: LEADER
I20260924 16:02:01.031349 28402 consensus_queue.cc:237] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:02:01.033716 27789 catalog_manager.cc:5495] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8f55ec33d6c418f97c5180d7e6c0e98 (127.25.109.2). New cstate: current_term: 1 leader_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } health_report { overall_health: UNKNOWN } } }
I20260924 16:02:01.057484 28410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:01.053861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48964 (local address 127.25.109.2:46761)
0924 16:02:01.054135 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:01.054142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:01.054154 (+ 12us) server_negotiation.cc:407] Connection header received
0924 16:02:01.054226 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:01.054228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:01.054281 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:01.054384 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:02:01.054623 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:01.055174 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:01.055971 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:01.056208 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:01.056284 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:01.056581 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:02:01.057052 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:02:01.057139 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:01.057224 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 16:02:01.057271 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":125,"threads_started":1}
I20260924 16:02:01.067816 28406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:01.065025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:54441 (local address 127.25.109.3:46755)
0924 16:02:01.065162 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:01.065165 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:01.065178 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:02:01.065246 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:01.065249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:01.065295 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:01.065359 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:02:01.065582 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:01.066069 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:01.066894 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:01.067106 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:01.067182 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:01.067432 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:02:01.067501 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:02:01.067553 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:01.067659 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 16:02:01.067707 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 16:02:01.067824 28413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:01.064776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:46755 (local address 127.25.109.2:54441)
0924 16:02:01.065084 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:01.065098 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:01.065175 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:01.065385 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:01.065386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:01.065392 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:02:01.065487 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:01.065492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:01.066173 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:01.066176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:01.066794 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:01.066802 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:01.067204 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:01.067207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:01.067274 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:01.067317 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:02:01.067576 (+ 259us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:02:01.067580 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 16:02:01.067630 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 16:02:01.067694 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":108,"threads_started":1}
I20260924 16:02:01.074501 27951 raft_consensus.cc:1261] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P d63f57e80946448f877730a1dd865a7d [term 1 FOLLOWER]: Refusing update from remote peer a8f55ec33d6c418f97c5180d7e6c0e98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:02:01.074630 28234 raft_consensus.cc:1261] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P 94fcab817cd443e1b3b35d5e4d009699 [term 1 FOLLOWER]: Refusing update from remote peer a8f55ec33d6c418f97c5180d7e6c0e98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:02:01.075037 28407 consensus_queue.cc:1035] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:01.075270 28402 consensus_queue.cc:1035] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:01.092823 28410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:01.089685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:57949 (local address 127.25.109.2:46761)
0924 16:02:01.089920 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:01.089926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:01.089940 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:02:01.089985 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:01.089989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:01.090037 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:01.090109 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:02:01.090348 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:01.090836 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:01.091721 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:01.091998 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:01.092113 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:01.092345 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:02:01.092450 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:02:01.092531 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:01.092627 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 16:02:01.092686 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260924 16:02:01.092860 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:01.089437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:46761 (local address 127.25.109.3:57949)
0924 16:02:01.089741 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:01.089757 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:01.089854 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:01.090151 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:01.090155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:01.090167 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:02:01.090272 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:01.090277 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:01.090925 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:01.090927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:01.091594 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:01.091607 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:01.092117 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:01.092121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:01.092208 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:01.092244 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:02:01.092534 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:02:01.092540 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 16:02:01.092616 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 16:02:01.092712 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":98,"threads_started":1}
I20260924 16:02:01.116021 27789 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:02:01.116168 27789 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:02:01.118347 27789 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0cdf45660a5941b28125b81a6b3d0a6e: Sending DeleteTablet for 3 replicas of tablet d4b72eb4ae214cb5ab2ca7aba9b3e854
I20260924 16:02:01.118923 27930 tablet_service.cc:1514] Processing DeleteTablet for tablet d4b72eb4ae214cb5ab2ca7aba9b3e854 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:01 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:44344
I20260924 16:02:01.118984 28072 tablet_service.cc:1514] Processing DeleteTablet for tablet d4b72eb4ae214cb5ab2ca7aba9b3e854 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:01 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:48956
I20260924 16:02:01.119139 28384 tablet_replica.cc:330] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98: stopping tablet replica
I20260924 16:02:01.119141 28385 tablet_replica.cc:330] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P d63f57e80946448f877730a1dd865a7d: stopping tablet replica
I20260924 16:02:01.119261 28384 raft_consensus.cc:2229] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:02:01.119270 28385 raft_consensus.cc:2229] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P d63f57e80946448f877730a1dd865a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:01.119380 28385 raft_consensus.cc:2258] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P d63f57e80946448f877730a1dd865a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:01.119581 28384 raft_consensus.cc:2258] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:01.120217 28385 ts_tablet_manager.cc:1905] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P d63f57e80946448f877730a1dd865a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:01.120328 28384 ts_tablet_manager.cc:1905] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:01.120332 27789 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:02:01.120775 27789 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:02:01.122011 28213 tablet_service.cc:1514] Processing DeleteTablet for tablet d4b72eb4ae214cb5ab2ca7aba9b3e854 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:01 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:47012
I20260924 16:02:01.122223 28386 tablet_replica.cc:330] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P 94fcab817cd443e1b3b35d5e4d009699: stopping tablet replica
I20260924 16:02:01.122347 28386 raft_consensus.cc:2229] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P 94fcab817cd443e1b3b35d5e4d009699 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:01.122480 28386 raft_consensus.cc:2258] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P 94fcab817cd443e1b3b35d5e4d009699 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:01.123026 28384 ts_tablet_manager.cc:1918] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:02:01.123028 28385 ts_tablet_manager.cc:1918] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P d63f57e80946448f877730a1dd865a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:02:01.123106 28385 log.cc:1196] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P d63f57e80946448f877730a1dd865a7d: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-0/wal/wals/d4b72eb4ae214cb5ab2ca7aba9b3e854
I20260924 16:02:01.123104 28384 log.cc:1196] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-1/wal/wals/d4b72eb4ae214cb5ab2ca7aba9b3e854
I20260924 16:02:01.123287 28386 ts_tablet_manager.cc:1905] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P 94fcab817cd443e1b3b35d5e4d009699: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:01.123381 28385 ts_tablet_manager.cc:1939] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P d63f57e80946448f877730a1dd865a7d: Deleting consensus metadata
I20260924 16:02:01.123385 28384 ts_tablet_manager.cc:1939] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P a8f55ec33d6c418f97c5180d7e6c0e98: Deleting consensus metadata
I20260924 16:02:01.124080 27773 catalog_manager.cc:4841] TS a8f55ec33d6c418f97c5180d7e6c0e98 (127.25.109.2:46761): tablet d4b72eb4ae214cb5ab2ca7aba9b3e854 (table test-table [id=d43c55f435764c68bcbd0414f8fbb983]) successfully deleted
I20260924 16:02:01.124543 27776 catalog_manager.cc:4841] TS d63f57e80946448f877730a1dd865a7d (127.25.109.1:37237): tablet d4b72eb4ae214cb5ab2ca7aba9b3e854 (table test-table [id=d43c55f435764c68bcbd0414f8fbb983]) successfully deleted
I20260924 16:02:01.125994 28386 ts_tablet_manager.cc:1918] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P 94fcab817cd443e1b3b35d5e4d009699: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:02:01.126065 28386 log.cc:1196] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P 94fcab817cd443e1b3b35d5e4d009699: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-2/wal/wals/d4b72eb4ae214cb5ab2ca7aba9b3e854
I20260924 16:02:01.126331 28386 ts_tablet_manager.cc:1939] T d4b72eb4ae214cb5ab2ca7aba9b3e854 P 94fcab817cd443e1b3b35d5e4d009699: Deleting consensus metadata
I20260924 16:02:01.127228 27774 catalog_manager.cc:4841] TS 94fcab817cd443e1b3b35d5e4d009699 (127.25.109.3:46755): tablet d4b72eb4ae214cb5ab2ca7aba9b3e854 (table test-table [id=d43c55f435764c68bcbd0414f8fbb983]) successfully deleted
I20260924 16:02:01.128472 27930 tablet_service.cc:1467] Processing CreateTablet for tablet 1bde0d63a63e44779789abd989253082 (DEFAULT_TABLE table=test-table [id=96a9ff805663473786f88e96f92ab375]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:01.128487 28072 tablet_service.cc:1467] Processing CreateTablet for tablet 1bde0d63a63e44779789abd989253082 (DEFAULT_TABLE table=test-table [id=96a9ff805663473786f88e96f92ab375]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:01.128731 28072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bde0d63a63e44779789abd989253082. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:01.128731 27930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bde0d63a63e44779789abd989253082. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:01.130888 28396 tablet_bootstrap.cc:492] T 1bde0d63a63e44779789abd989253082 P a8f55ec33d6c418f97c5180d7e6c0e98: Bootstrap starting.
I20260924 16:02:01.130937 28395 tablet_bootstrap.cc:492] T 1bde0d63a63e44779789abd989253082 P d63f57e80946448f877730a1dd865a7d: Bootstrap starting.
I20260924 16:02:01.131747 28395 tablet_bootstrap.cc:654] T 1bde0d63a63e44779789abd989253082 P d63f57e80946448f877730a1dd865a7d: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:01.131866 28396 tablet_bootstrap.cc:654] T 1bde0d63a63e44779789abd989253082 P a8f55ec33d6c418f97c5180d7e6c0e98: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:01.132862 28395 tablet_bootstrap.cc:492] T 1bde0d63a63e44779789abd989253082 P d63f57e80946448f877730a1dd865a7d: No bootstrap required, opened a new log
I20260924 16:02:01.132893 28213 tablet_service.cc:1467] Processing CreateTablet for tablet 1bde0d63a63e44779789abd989253082 (DEFAULT_TABLE table=test-table [id=96a9ff805663473786f88e96f92ab375]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:01.132949 28395 ts_tablet_manager.cc:1397] T 1bde0d63a63e44779789abd989253082 P d63f57e80946448f877730a1dd865a7d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 16:02:01.133152 28213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bde0d63a63e44779789abd989253082. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:01.133384 28395 raft_consensus.cc:348] T 1bde0d63a63e44779789abd989253082 P d63f57e80946448f877730a1dd865a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:02:01.133500 28395 raft_consensus.cc:374] T 1bde0d63a63e44779789abd989253082 P d63f57e80946448f877730a1dd865a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:01.133533 28395 raft_consensus.cc:729] T 1bde0d63a63e44779789abd989253082 P d63f57e80946448f877730a1dd865a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d63f57e80946448f877730a1dd865a7d, State: Initialized, Role: FOLLOWER
I20260924 16:02:01.133666 28395 consensus_queue.cc:260] T 1bde0d63a63e44779789abd989253082 P d63f57e80946448f877730a1dd865a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:02:01.133908 28395 ts_tablet_manager.cc:1428] T 1bde0d63a63e44779789abd989253082 P d63f57e80946448f877730a1dd865a7d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 16:02:01.135202 28398 tablet_bootstrap.cc:492] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699: Bootstrap starting.
I20260924 16:02:01.136072 28398 tablet_bootstrap.cc:654] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:01.137025 28396 tablet_bootstrap.cc:492] T 1bde0d63a63e44779789abd989253082 P a8f55ec33d6c418f97c5180d7e6c0e98: No bootstrap required, opened a new log
I20260924 16:02:01.137116 28396 ts_tablet_manager.cc:1397] T 1bde0d63a63e44779789abd989253082 P a8f55ec33d6c418f97c5180d7e6c0e98: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260924 16:02:01.137552 28396 raft_consensus.cc:348] T 1bde0d63a63e44779789abd989253082 P a8f55ec33d6c418f97c5180d7e6c0e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:02:01.137665 28396 raft_consensus.cc:374] T 1bde0d63a63e44779789abd989253082 P a8f55ec33d6c418f97c5180d7e6c0e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:01.137742 28396 raft_consensus.cc:729] T 1bde0d63a63e44779789abd989253082 P a8f55ec33d6c418f97c5180d7e6c0e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8f55ec33d6c418f97c5180d7e6c0e98, State: Initialized, Role: FOLLOWER
I20260924 16:02:01.137864 28396 consensus_queue.cc:260] T 1bde0d63a63e44779789abd989253082 P a8f55ec33d6c418f97c5180d7e6c0e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:02:01.138134 28396 ts_tablet_manager.cc:1428] T 1bde0d63a63e44779789abd989253082 P a8f55ec33d6c418f97c5180d7e6c0e98: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 16:02:01.139037 28398 tablet_bootstrap.cc:492] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699: No bootstrap required, opened a new log
I20260924 16:02:01.139128 28398 ts_tablet_manager.cc:1397] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260924 16:02:01.139532 28398 raft_consensus.cc:348] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:02:01.139644 28398 raft_consensus.cc:374] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:01.139674 28398 raft_consensus.cc:729] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94fcab817cd443e1b3b35d5e4d009699, State: Initialized, Role: FOLLOWER
I20260924 16:02:01.139792 28398 consensus_queue.cc:260] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:02:01.140051 28398 ts_tablet_manager.cc:1428] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 16:02:01.220659 28366 raft_consensus.cc:482] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:02:01.220829 28366 raft_consensus.cc:504] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:02:01.221341 28366 leader_election.cc:288] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d63f57e80946448f877730a1dd865a7d (127.25.109.1:37237), a8f55ec33d6c418f97c5180d7e6c0e98 (127.25.109.2:46761)
I20260924 16:02:01.221863 27951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bde0d63a63e44779789abd989253082" candidate_uuid: "94fcab817cd443e1b3b35d5e4d009699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d63f57e80946448f877730a1dd865a7d" is_pre_election: true
I20260924 16:02:01.221895 28093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bde0d63a63e44779789abd989253082" candidate_uuid: "94fcab817cd443e1b3b35d5e4d009699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" is_pre_election: true
I20260924 16:02:01.222033 27951 raft_consensus.cc:2454] T 1bde0d63a63e44779789abd989253082 P d63f57e80946448f877730a1dd865a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94fcab817cd443e1b3b35d5e4d009699 in term 0.
I20260924 16:02:01.222062 28093 raft_consensus.cc:2454] T 1bde0d63a63e44779789abd989253082 P a8f55ec33d6c418f97c5180d7e6c0e98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94fcab817cd443e1b3b35d5e4d009699 in term 0.
I20260924 16:02:01.222397 28166 leader_election.cc:302] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94fcab817cd443e1b3b35d5e4d009699, d63f57e80946448f877730a1dd865a7d; no voters:
I20260924 16:02:01.222692 28366 raft_consensus.cc:2781] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:02:01.222766 28366 raft_consensus.cc:482] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:02:01.222824 28366 raft_consensus.cc:3037] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:01.223893 28366 raft_consensus.cc:504] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:02:01.224321 28366 leader_election.cc:288] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699 [CANDIDATE]: Term 1 election: Requested vote from peers d63f57e80946448f877730a1dd865a7d (127.25.109.1:37237), a8f55ec33d6c418f97c5180d7e6c0e98 (127.25.109.2:46761)
I20260924 16:02:01.224730 28093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bde0d63a63e44779789abd989253082" candidate_uuid: "94fcab817cd443e1b3b35d5e4d009699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98"
I20260924 16:02:01.224731 27951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bde0d63a63e44779789abd989253082" candidate_uuid: "94fcab817cd443e1b3b35d5e4d009699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d63f57e80946448f877730a1dd865a7d"
I20260924 16:02:01.224860 27951 raft_consensus.cc:3037] T 1bde0d63a63e44779789abd989253082 P d63f57e80946448f877730a1dd865a7d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:01.224876 28093 raft_consensus.cc:3037] T 1bde0d63a63e44779789abd989253082 P a8f55ec33d6c418f97c5180d7e6c0e98 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:01.225791 28093 raft_consensus.cc:2454] T 1bde0d63a63e44779789abd989253082 P a8f55ec33d6c418f97c5180d7e6c0e98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94fcab817cd443e1b3b35d5e4d009699 in term 1.
I20260924 16:02:01.226130 28164 leader_election.cc:302] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94fcab817cd443e1b3b35d5e4d009699, a8f55ec33d6c418f97c5180d7e6c0e98; no voters:
I20260924 16:02:01.226140 27951 raft_consensus.cc:2454] T 1bde0d63a63e44779789abd989253082 P d63f57e80946448f877730a1dd865a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94fcab817cd443e1b3b35d5e4d009699 in term 1.
I20260924 16:02:01.226328 28366 raft_consensus.cc:2781] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:01.226414 28366 raft_consensus.cc:686] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699 [term 1 LEADER]: Becoming Leader. State: Replica: 94fcab817cd443e1b3b35d5e4d009699, State: Running, Role: LEADER
I20260924 16:02:01.226573 28366 consensus_queue.cc:237] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } }
I20260924 16:02:01.228683 27789 catalog_manager.cc:5495] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94fcab817cd443e1b3b35d5e4d009699 (127.25.109.3). New cstate: current_term: 1 leader_uuid: "94fcab817cd443e1b3b35d5e4d009699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94fcab817cd443e1b3b35d5e4d009699" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 46755 } health_report { overall_health: HEALTHY } } }
I20260924 16:02:01.244612 28406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:01.241449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47044 (local address 127.25.109.3:46755)
0924 16:02:01.241770 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:01.241774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:01.241783 (+ 9us) server_negotiation.cc:407] Connection header received
0924 16:02:01.241820 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:01.241822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:01.241859 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:01.241943 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:02:01.242196 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:01.242706 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:01.243547 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:01.243766 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:01.243877 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:01.244141 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:02:01.244214 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:02:01.244295 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:01.244391 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 16:02:01.244464 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249}
I20260924 16:02:01.251765 27951 raft_consensus.cc:1261] T 1bde0d63a63e44779789abd989253082 P d63f57e80946448f877730a1dd865a7d [term 1 FOLLOWER]: Refusing update from remote peer 94fcab817cd443e1b3b35d5e4d009699: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:02:01.252050 28093 raft_consensus.cc:1261] T 1bde0d63a63e44779789abd989253082 P a8f55ec33d6c418f97c5180d7e6c0e98 [term 1 FOLLOWER]: Refusing update from remote peer 94fcab817cd443e1b3b35d5e4d009699: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:02:01.252200 28366 consensus_queue.cc:1035] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d63f57e80946448f877730a1dd865a7d" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:01.252487 28350 consensus_queue.cc:1035] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8f55ec33d6c418f97c5180d7e6c0e98" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 16:02:01.267716 28364 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 16:02:02.818095 27790 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:02:02.818252 27790 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:02:02.820468 27790 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0cdf45660a5941b28125b81a6b3d0a6e: Sending DeleteTablet for 3 replicas of tablet 1bde0d63a63e44779789abd989253082
I20260924 16:02:02.820979 27930 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bde0d63a63e44779789abd989253082 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:02 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:44344
I20260924 16:02:02.821141 28072 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bde0d63a63e44779789abd989253082 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:02 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:48956
I20260924 16:02:02.821326 28446 tablet_replica.cc:330] T 1bde0d63a63e44779789abd989253082 P d63f57e80946448f877730a1dd865a7d: stopping tablet replica
I20260924 16:02:02.821471 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 27863
I20260924 16:02:02.821502 28446 raft_consensus.cc:2229] T 1bde0d63a63e44779789abd989253082 P d63f57e80946448f877730a1dd865a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:02.822005 28212 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bde0d63a63e44779789abd989253082 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:02 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:47012
I20260924 16:02:02.822340 28448 tablet_replica.cc:330] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699: stopping tablet replica
I20260924 16:02:02.822485 28448 raft_consensus.cc:2229] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:02:02.822664 28448 raft_consensus.cc:2258] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:02.821523 28447 tablet_replica.cc:330] T 1bde0d63a63e44779789abd989253082 P a8f55ec33d6c418f97c5180d7e6c0e98: stopping tablet replica
I20260924 16:02:02.822929 28447 raft_consensus.cc:2229] T 1bde0d63a63e44779789abd989253082 P a8f55ec33d6c418f97c5180d7e6c0e98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:02.823101 28447 raft_consensus.cc:2258] T 1bde0d63a63e44779789abd989253082 P a8f55ec33d6c418f97c5180d7e6c0e98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:02.823391 28448 ts_tablet_manager.cc:1905] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:02.823837 28447 ts_tablet_manager.cc:1905] T 1bde0d63a63e44779789abd989253082 P a8f55ec33d6c418f97c5180d7e6c0e98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:02.826156 28447 ts_tablet_manager.cc:1918] T 1bde0d63a63e44779789abd989253082 P a8f55ec33d6c418f97c5180d7e6c0e98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 16:02:02.826258 28447 log.cc:1196] T 1bde0d63a63e44779789abd989253082 P a8f55ec33d6c418f97c5180d7e6c0e98: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-1/wal/wals/1bde0d63a63e44779789abd989253082
I20260924 16:02:02.826499 28447 ts_tablet_manager.cc:1939] T 1bde0d63a63e44779789abd989253082 P a8f55ec33d6c418f97c5180d7e6c0e98: Deleting consensus metadata
I20260924 16:02:02.826862 28448 ts_tablet_manager.cc:1918] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 16:02:02.826946 28448 log.cc:1196] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265708917108-26036-0/minicluster-data/ts-2/wal/wals/1bde0d63a63e44779789abd989253082
I20260924 16:02:02.827188 27773 catalog_manager.cc:4841] TS a8f55ec33d6c418f97c5180d7e6c0e98 (127.25.109.2:46761): tablet 1bde0d63a63e44779789abd989253082 (table test-table [id=96a9ff805663473786f88e96f92ab375]) successfully deleted
I20260924 16:02:02.827250 28448 ts_tablet_manager.cc:1939] T 1bde0d63a63e44779789abd989253082 P 94fcab817cd443e1b3b35d5e4d009699: Deleting consensus metadata
I20260924 16:02:02.827925 27774 catalog_manager.cc:4841] TS 94fcab817cd443e1b3b35d5e4d009699 (127.25.109.3:46755): tablet 1bde0d63a63e44779789abd989253082 (table test-table [id=96a9ff805663473786f88e96f92ab375]) successfully deleted
W20260924 16:02:02.831202 27776 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.109.1:37237 (error 108)
I20260924 16:02:02.831419 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 28006
I20260924 16:02:02.832616 28450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:02.832138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:37237 (local address 127.0.0.1:44364)
0924 16:02:02.832373 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:02.832493 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.1:37237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":94,"threads_started":1}
W20260924 16:02:02.832935 27776 catalog_manager.cc:4584] TS d63f57e80946448f877730a1dd865a7d (127.25.109.1:37237): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1bde0d63a63e44779789abd989253082: Network error: Client connection negotiation failed: client connection to 127.25.109.1:37237: connect: Connection refused (error 111)
I20260924 16:02:02.839039 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 28148
I20260924 16:02:02.846240 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 27756
2026-09-24T16:02:02Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6325 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:02:02 dist-test-slave-n106 krb5kdc[28453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:02:02 dist-test-slave-n106 krb5kdc[28453](info): set up 2 sockets
Sep 24 16:02:02 dist-test-slave-n106 krb5kdc[28453](info): commencing operation
krb5kdc: starting...
W20260924 16:02:03.901489 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:02:03 dist-test-slave-n106 krb5kdc[28453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:02:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:02:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
I20260924 16:02:04.019352 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:39397
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:39397
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:04.127652 28466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:04.127961 28466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:04.128046 28466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:04.132468 28466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:02:04.132582 28466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:04.132627 28466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:04.132690 28466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:02:04.132726 28466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:02:04.138025 28466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:39397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:39397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:04.139137 28466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:04.140177 28466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:04.149259 28474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:04.149240 28472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:04.149240 28471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:04.149293 28466 server_base.cc:1034] running on GCE node
I20260924 16:02:04.150132 28466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:04.151212 28466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:04.152437 28466 hybrid_clock.cc:648] HybridClock initialized: now 1790265724152386 us; error 36 us; skew 500 ppm
Sep 24 16:02:04 dist-test-slave-n106 krb5kdc[28453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265724, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:04.155107 28466 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:02:04.156090 28466 webserver.cc:460] Webserver started at http://127.25.109.62:46061/ using document root <none> and password file <none>
I20260924 16:02:04.156718 28466 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:04.156807 28466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:04.157048 28466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:04.159053 28466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "b493108db9d74bd6b25451bf95637ab4"
format_stamp: "Formatted at 2026-09-24 16:02:04 on dist-test-slave-n106"
server_key: "45cf24aabf1858ecdac8cd12ad60db3e"
server_key_iv: "5a773760c6cd9e3299daf73b90eb6ad7"
server_key_version: "encryptionkey@0"
I20260924 16:02:04.159631 28466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "b493108db9d74bd6b25451bf95637ab4"
format_stamp: "Formatted at 2026-09-24 16:02:04 on dist-test-slave-n106"
server_key: "45cf24aabf1858ecdac8cd12ad60db3e"
server_key_iv: "5a773760c6cd9e3299daf73b90eb6ad7"
server_key_version: "encryptionkey@0"
I20260924 16:02:04.163717 28466 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 16:02:04.166224 28481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:04.167445 28466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 16:02:04.167608 28466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "b493108db9d74bd6b25451bf95637ab4"
format_stamp: "Formatted at 2026-09-24 16:02:04 on dist-test-slave-n106"
server_key: "45cf24aabf1858ecdac8cd12ad60db3e"
server_key_iv: "5a773760c6cd9e3299daf73b90eb6ad7"
server_key_version: "encryptionkey@0"
I20260924 16:02:04.167723 28466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:04.189740 28466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:04.192667 28466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:04.192873 28466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:04.199864 28466 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:39397
I20260924 16:02:04.199891 28534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:39397 every 8 connection(s)
I20260924 16:02:04.200824 28466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:02:04.203552 28535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:04.205396 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 28466
I20260924 16:02:04.205485 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:02:04.205782 26036 external_mini_cluster.cc:1428] Setting key 6fe50e80953272c6f0e2e738874af114
I20260924 16:02:04.209970 28535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b493108db9d74bd6b25451bf95637ab4: Bootstrap starting.
I20260924 16:02:04.212252 28535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b493108db9d74bd6b25451bf95637ab4: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:04.213152 28535 log.cc:824] T 00000000000000000000000000000000 P b493108db9d74bd6b25451bf95637ab4: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:02:04 dist-test-slave-n106 krb5kdc[28453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:02:04.215356 28535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b493108db9d74bd6b25451bf95637ab4: No bootstrap required, opened a new log
I20260924 16:02:04.218961 28535 raft_consensus.cc:348] T 00000000000000000000000000000000 P b493108db9d74bd6b25451bf95637ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b493108db9d74bd6b25451bf95637ab4" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 39397 } }
I20260924 16:02:04.219285 28535 raft_consensus.cc:374] T 00000000000000000000000000000000 P b493108db9d74bd6b25451bf95637ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:04.219367 28535 raft_consensus.cc:729] T 00000000000000000000000000000000 P b493108db9d74bd6b25451bf95637ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b493108db9d74bd6b25451bf95637ab4, State: Initialized, Role: FOLLOWER
I20260924 16:02:04.219810 28535 consensus_queue.cc:260] T 00000000000000000000000000000000 P b493108db9d74bd6b25451bf95637ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b493108db9d74bd6b25451bf95637ab4" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 39397 } }
I20260924 16:02:04.219930 28535 raft_consensus.cc:388] T 00000000000000000000000000000000 P b493108db9d74bd6b25451bf95637ab4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:02:04.220017 28535 raft_consensus.cc:482] T 00000000000000000000000000000000 P b493108db9d74bd6b25451bf95637ab4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:02:04.220151 28535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b493108db9d74bd6b25451bf95637ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:04.220180 28538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:04.207247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49932 (local address 127.25.109.62:39397)
0924 16:02:04.207695 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:04.207711 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:04.207747 (+ 36us) server_negotiation.cc:407] Connection header received
0924 16:02:04.208388 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:04.208459 (+ 71us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:04.208812 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:04.209152 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:04.209986 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:04.211006 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:04.211887 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:04.212401 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:04.212581 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:04.215297 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:04.215302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:04.215315 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:04.215342 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:04.217118 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:04.217580 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:04.217586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:04.217591 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:04.217699 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:04.217985 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:04.217989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:04.217991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:04.218378 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:04.218586 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:04.218898 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 16:02:04.219190 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":124,"threads_started":1}
I20260924 16:02:04.221089 28535 raft_consensus.cc:504] T 00000000000000000000000000000000 P b493108db9d74bd6b25451bf95637ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b493108db9d74bd6b25451bf95637ab4" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 39397 } }
I20260924 16:02:04.221469 28535 leader_election.cc:302] T 00000000000000000000000000000000 P b493108db9d74bd6b25451bf95637ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b493108db9d74bd6b25451bf95637ab4; no voters:
I20260924 16:02:04.221796 28535 leader_election.cc:288] T 00000000000000000000000000000000 P b493108db9d74bd6b25451bf95637ab4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:02:04.221912 28540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b493108db9d74bd6b25451bf95637ab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:04.222133 28540 raft_consensus.cc:686] T 00000000000000000000000000000000 P b493108db9d74bd6b25451bf95637ab4 [term 1 LEADER]: Becoming Leader. State: Replica: b493108db9d74bd6b25451bf95637ab4, State: Running, Role: LEADER
I20260924 16:02:04.222481 28540 consensus_queue.cc:237] T 00000000000000000000000000000000 P b493108db9d74bd6b25451bf95637ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b493108db9d74bd6b25451bf95637ab4" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 39397 } }
I20260924 16:02:04.223034 28535 sys_catalog.cc:564] T 00000000000000000000000000000000 P b493108db9d74bd6b25451bf95637ab4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:02:04.223737 28540 sys_catalog.cc:455] T 00000000000000000000000000000000 P b493108db9d74bd6b25451bf95637ab4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b493108db9d74bd6b25451bf95637ab4. Latest consensus state: current_term: 1 leader_uuid: "b493108db9d74bd6b25451bf95637ab4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b493108db9d74bd6b25451bf95637ab4" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 39397 } } }
I20260924 16:02:04.223861 28540 sys_catalog.cc:458] T 00000000000000000000000000000000 P b493108db9d74bd6b25451bf95637ab4 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:04.224287 28542 sys_catalog.cc:455] T 00000000000000000000000000000000 P b493108db9d74bd6b25451bf95637ab4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b493108db9d74bd6b25451bf95637ab4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b493108db9d74bd6b25451bf95637ab4" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 39397 } } }
I20260924 16:02:04.224391 28542 sys_catalog.cc:458] T 00000000000000000000000000000000 P b493108db9d74bd6b25451bf95637ab4 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:04.224763 28547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:02:04.227495 28547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:02:04.232755 28547 catalog_manager.cc:1347] Generated new cluster ID: 36f14e7d8b9d453782c49e254dc4079b
I20260924 16:02:04.232820 28547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:02:04.253415 28547 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:02:04.254783 28547 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:02:04.262229 28547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b493108db9d74bd6b25451bf95637ab4: Generated new TSK 0
I20260924 16:02:04.263036 28547 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
I20260924 16:02:04.296254 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:0
--local_ip_for_outbound_sockets=127.25.109.1
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:39397
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:04.407891 28562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:04.408140 28562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:04.408182 28562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:04.412218 28562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:04.412268 28562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:04.412325 28562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:02:04.418051 28562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:39397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:04.419106 28562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:04.420198 28562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:04.432077 28567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:04.432061 28568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:04.432385 28562 server_base.cc:1034] running on GCE node
W20260924 16:02:04.432086 28570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:04.432852 28562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:04.433511 28562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:04.434697 28562 hybrid_clock.cc:648] HybridClock initialized: now 1790265724434665 us; error 48 us; skew 500 ppm
Sep 24 16:02:04 dist-test-slave-n106 krb5kdc[28453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265724, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:04.437470 28562 init.cc:377] Logged in from keytab as kudu/127.25.109.1@KRBTEST.COM (short username kudu)
I20260924 16:02:04.438339 28562 webserver.cc:460] Webserver started at http://127.25.109.1:39717/ using document root <none> and password file <none>
I20260924 16:02:04.438928 28562 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:04.439023 28562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:04.439263 28562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:04.441412 28562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/data/instance:
uuid: "2334298c4b934e3d833f2b66682ca86b"
format_stamp: "Formatted at 2026-09-24 16:02:04 on dist-test-slave-n106"
server_key: "c21205869ed16fcf13b94d67497c5da2"
server_key_iv: "7bc482230c921f3d313b50377c82a654"
server_key_version: "encryptionkey@0"
I20260924 16:02:04.442060 28562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance:
uuid: "2334298c4b934e3d833f2b66682ca86b"
format_stamp: "Formatted at 2026-09-24 16:02:04 on dist-test-slave-n106"
server_key: "c21205869ed16fcf13b94d67497c5da2"
server_key_iv: "7bc482230c921f3d313b50377c82a654"
server_key_version: "encryptionkey@0"
I20260924 16:02:04.445894 28562 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:02:04.448294 28577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:04.449563 28562 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:02:04.449712 28562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "2334298c4b934e3d833f2b66682ca86b"
format_stamp: "Formatted at 2026-09-24 16:02:04 on dist-test-slave-n106"
server_key: "c21205869ed16fcf13b94d67497c5da2"
server_key_iv: "7bc482230c921f3d313b50377c82a654"
server_key_version: "encryptionkey@0"
I20260924 16:02:04.449828 28562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:04.466120 28562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:04.469432 28562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:04.469648 28562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:04.470249 28562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:04.471165 28562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:04.471240 28562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:04.471318 28562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:04.471369 28562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:04.480810 28562 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:33161
I20260924 16:02:04.480837 28693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:33161 every 8 connection(s)
I20260924 16:02:04.481730 28562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:02:04.482743 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 28562
I20260924 16:02:04.482862 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance
I20260924 16:02:04.483184 26036 external_mini_cluster.cc:1428] Setting key e8382facb4fb45e53993674d63567788
Sep 24 16:02:04 dist-test-slave-n106 krb5kdc[28453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265724, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.2@KRBTEST.COM" created.
I20260924 16:02:04.495524 28538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:04.483663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:36237 (local address 127.25.109.62:39397)
0924 16:02:04.483799 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:04.483803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:04.484793 (+ 990us) server_negotiation.cc:407] Connection header received
0924 16:02:04.486068 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:04.486072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:04.486128 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:04.486208 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:04.487347 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:04.487890 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:04.489078 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:04.489315 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:04.489423 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:04.491886 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:04.491890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:04.491891 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:04.491915 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:04.493376 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:04.493876 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:04.493880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:04.493882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:04.493963 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:04.494348 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:04.494354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:04.494357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:04.494539 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:04.494682 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:04.495229 (+ 547us) server_negotiation.cc:299] Negotiation successful
0924 16:02:04.495350 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 16:02:04.496325 28696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:04.484144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39397 (local address 127.25.109.1:36237)
0924 16:02:04.484655 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:04.484698 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:04.485847 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:04.486382 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:04.486389 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:04.486734 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:04.487202 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:04.487216 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:04.488040 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:04.488044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:04.488946 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:04.488956 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:04.489461 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:04.489466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:04.489652 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:04.490280 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:04.490297 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:04.491717 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:04.493492 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:04.493498 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:04.493519 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:04.493770 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:04.494073 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:04.494076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:04.494078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:04.494249 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:04.494683 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:04.494691 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:04.494983 (+ 292us) client_negotiation.cc:769] Sending connection context
0924 16:02:04.495189 (+ 206us) client_negotiation.cc:240] Negotiation successful
0924 16:02:04.495495 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":135,"threads_started":1}
I20260924 16:02:04.497594 28694 heartbeater.cc:347] Connected to a master server at 127.25.109.62:39397
I20260924 16:02:04.497865 28694 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:04.498469 28694 heartbeater.cc:511] Master 127.25.109.62:39397 requested a full tablet report, sending...
I20260924 16:02:04.500106 28499 ts_manager.cc:194] Registered new tserver with Master: 2334298c4b934e3d833f2b66682ca86b (127.25.109.1:33161)
I20260924 16:02:04.501391 28499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.1@KRBTEST.COM'} at 127.25.109.1:36237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
I20260924 16:02:04.510900 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:0
--local_ip_for_outbound_sockets=127.25.109.2
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:39397
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:04.620522 28699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:04.620817 28699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:04.620901 28699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:04.625141 28699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:04.625216 28699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:04.625305 28699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:02:04.630522 28699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:39397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:04.631577 28699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:04.632643 28699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:04.642442 28707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:04.642452 28705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:04.642454 28704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:04.642761 28699 server_base.cc:1034] running on GCE node
I20260924 16:02:04.643285 28699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:04.643957 28699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:04.645143 28699 hybrid_clock.cc:648] HybridClock initialized: now 1790265724645108 us; error 47 us; skew 500 ppm
Sep 24 16:02:04 dist-test-slave-n106 krb5kdc[28453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265724, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:04.647984 28699 init.cc:377] Logged in from keytab as kudu/127.25.109.2@KRBTEST.COM (short username kudu)
I20260924 16:02:04.648960 28699 webserver.cc:460] Webserver started at http://127.25.109.2:46829/ using document root <none> and password file <none>
I20260924 16:02:04.649526 28699 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:04.649629 28699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:04.649864 28699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:04.651893 28699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/data/instance:
uuid: "c6a2b704ede445e79070dd22bc52f246"
format_stamp: "Formatted at 2026-09-24 16:02:04 on dist-test-slave-n106"
server_key: "202a40a373925860173a39a07ca41dda"
server_key_iv: "4401c437c76e8c1962181cd20be935e6"
server_key_version: "encryptionkey@0"
I20260924 16:02:04.652544 28699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance:
uuid: "c6a2b704ede445e79070dd22bc52f246"
format_stamp: "Formatted at 2026-09-24 16:02:04 on dist-test-slave-n106"
server_key: "202a40a373925860173a39a07ca41dda"
server_key_iv: "4401c437c76e8c1962181cd20be935e6"
server_key_version: "encryptionkey@0"
I20260924 16:02:04.656390 28699 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:02:04.658927 28714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:04.660128 28699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 16:02:04.660270 28699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "c6a2b704ede445e79070dd22bc52f246"
format_stamp: "Formatted at 2026-09-24 16:02:04 on dist-test-slave-n106"
server_key: "202a40a373925860173a39a07ca41dda"
server_key_iv: "4401c437c76e8c1962181cd20be935e6"
server_key_version: "encryptionkey@0"
I20260924 16:02:04.660385 28699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:04.685279 28699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:04.688673 28699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:04.688871 28699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:04.689460 28699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:04.690299 28699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:04.690373 28699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:04.690443 28699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:04.690492 28699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:04.699481 28699 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:37249
I20260924 16:02:04.699509 28829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:37249 every 8 connection(s)
I20260924 16:02:04.700332 28699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:02:04.707113 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 28699
I20260924 16:02:04.707231 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance
I20260924 16:02:04.707572 26036 external_mini_cluster.cc:1428] Setting key 0a006a8959b8724a3d10138a568e37f0
Sep 24 16:02:04 dist-test-slave-n106 krb5kdc[28453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265724, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:02:04.714039 28538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:04.702095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:53235 (local address 127.25.109.62:39397)
0924 16:02:04.702228 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:04.702232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:04.702983 (+ 751us) server_negotiation.cc:407] Connection header received
0924 16:02:04.704310 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:04.704313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:04.704369 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:04.704491 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:04.705619 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:04.706151 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:04.707533 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:04.707817 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:04.707906 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:04.710430 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:04.710434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:04.710436 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:04.710470 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:04.711892 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:04.712452 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:04.712457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:04.712463 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:04.712538 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:04.712910 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:04.712915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:04.712917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:04.713128 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:04.713240 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:04.713717 (+ 477us) server_negotiation.cc:299] Negotiation successful
0924 16:02:04.713841 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 16:02:04.714846 28832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:04.702399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39397 (local address 127.25.109.2:53235)
0924 16:02:04.702837 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:04.702879 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:04.704075 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:04.704673 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:04.704680 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:04.705010 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:04.705470 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:04.705484 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:04.706285 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:04.706294 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:04.707406 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:04.707419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:04.707982 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:04.707986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:04.708134 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:04.708724 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:04.708743 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:04.710265 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:04.712026 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:04.712032 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:04.712046 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:04.712306 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:04.712656 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:04.712659 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:04.712661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:04.712810 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:04.713241 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:04.713248 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:04.713480 (+ 232us) client_negotiation.cc:769] Sending connection context
0924 16:02:04.713669 (+ 189us) client_negotiation.cc:240] Negotiation successful
0924 16:02:04.713978 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":121,"threads_started":1}
I20260924 16:02:04.716040 28830 heartbeater.cc:347] Connected to a master server at 127.25.109.62:39397
I20260924 16:02:04.716298 28830 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.25.109.3@KRBTEST.COM; defaulting to no policy
I20260924 16:02:04.716861 28830 heartbeater.cc:511] Master 127.25.109.62:39397 requested a full tablet report, sending...
I20260924 16:02:04.717922 28499 ts_manager.cc:194] Registered new tserver with Master: c6a2b704ede445e79070dd22bc52f246 (127.25.109.2:37249)
I20260924 16:02:04.718523 28499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.2@KRBTEST.COM'} at 127.25.109.2:53235
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
I20260924 16:02:04.736160 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:0
--local_ip_for_outbound_sockets=127.25.109.3
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:39397
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:04.847489 28835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:04.847733 28835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:04.847774 28835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:04.851907 28835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:04.851958 28835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:04.852013 28835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:02:04.857371 28835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:39397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:04.858407 28835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:04.859459 28835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:04.869220 28843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:04.869288 28841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:04.869356 28835 server_base.cc:1034] running on GCE node
W20260924 16:02:04.869288 28840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:04.869941 28835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:04.870523 28835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:04.871726 28835 hybrid_clock.cc:648] HybridClock initialized: now 1790265724871707 us; error 26 us; skew 500 ppm
Sep 24 16:02:04 dist-test-slave-n106 krb5kdc[28453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265724, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:04.874399 28835 init.cc:377] Logged in from keytab as kudu/127.25.109.3@KRBTEST.COM (short username kudu)
I20260924 16:02:04.875214 28835 webserver.cc:460] Webserver started at http://127.25.109.3:40091/ using document root <none> and password file <none>
I20260924 16:02:04.875766 28835 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:04.875826 28835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:04.876060 28835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:04.878123 28835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/data/instance:
uuid: "9fd3858faf994afba13261a0bec4cec5"
format_stamp: "Formatted at 2026-09-24 16:02:04 on dist-test-slave-n106"
server_key: "b76a27be2a8733bc783b598bf101eaac"
server_key_iv: "1b06c2609a821b1d0f4af0de707cfa4f"
server_key_version: "encryptionkey@0"
I20260924 16:02:04.878693 28835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance:
uuid: "9fd3858faf994afba13261a0bec4cec5"
format_stamp: "Formatted at 2026-09-24 16:02:04 on dist-test-slave-n106"
server_key: "b76a27be2a8733bc783b598bf101eaac"
server_key_iv: "1b06c2609a821b1d0f4af0de707cfa4f"
server_key_version: "encryptionkey@0"
I20260924 16:02:04.882385 28835 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260924 16:02:04.884829 28850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:04.886019 28835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:02:04.886142 28835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "9fd3858faf994afba13261a0bec4cec5"
format_stamp: "Formatted at 2026-09-24 16:02:04 on dist-test-slave-n106"
server_key: "b76a27be2a8733bc783b598bf101eaac"
server_key_iv: "1b06c2609a821b1d0f4af0de707cfa4f"
server_key_version: "encryptionkey@0"
I20260924 16:02:04.886248 28835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:04.900502 28835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:04.903348 28835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:04.903570 28835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:04.904165 28835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:04.905001 28835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:04.905048 28835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:04.905119 28835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:04.905153 28835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:04.914112 28835 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:42191
I20260924 16:02:04.914153 28964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:42191 every 8 connection(s)
I20260924 16:02:04.914976 28835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:02:04.922351 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 28835
I20260924 16:02:04.922483 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance
I20260924 16:02:04.922827 26036 external_mini_cluster.cc:1428] Setting key 9d400d9400ad1996521173a1db2bc086
Sep 24 16:02:04 dist-test-slave-n106 krb5kdc[28453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265724, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:02:04.928697 28538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:04.916788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:46993 (local address 127.25.109.62:39397)
0924 16:02:04.916937 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:04.916941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:04.917746 (+ 805us) server_negotiation.cc:407] Connection header received
0924 16:02:04.919023 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:04.919026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:04.919085 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:04.919178 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:04.920308 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:04.920903 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:04.922045 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:04.922283 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:04.922398 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:04.925439 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:04.925442 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:04.925444 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:04.925467 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:04.926952 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:04.927386 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:04.927388 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:04.927389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:04.927463 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:04.927750 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:04.927752 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:04.927753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:04.927920 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:04.927996 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:04.928473 (+ 477us) server_negotiation.cc:299] Negotiation successful
0924 16:02:04.928579 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 16:02:04.929548 28967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:04.917156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39397 (local address 127.25.109.3:46993)
0924 16:02:04.917582 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:04.917623 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:04.918807 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:04.919349 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:04.919356 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:04.919697 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:04.920151 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:04.920163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:04.921033 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:04.921041 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:04.921908 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:04.921917 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:04.922436 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:04.922440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:04.922754 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:04.923395 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:04.923433 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:04.925293 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:04.927048 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:04.927052 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:04.927063 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:04.927302 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:04.927540 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:04.927542 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:04.927543 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:04.927672 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:04.928007 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:04.928011 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:04.928233 (+ 222us) client_negotiation.cc:769] Sending connection context
0924 16:02:04.928382 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 16:02:04.928683 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":99,"threads_started":1}
I20260924 16:02:04.930980 28965 heartbeater.cc:347] Connected to a master server at 127.25.109.62:39397
I20260924 16:02:04.931250 28965 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:04.931752 28965 heartbeater.cc:511] Master 127.25.109.62:39397 requested a full tablet report, sending...
I20260924 16:02:04.932776 28499 ts_manager.cc:194] Registered new tserver with Master: 9fd3858faf994afba13261a0bec4cec5 (127.25.109.3:42191)
I20260924 16:02:04.933331 28499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.3@KRBTEST.COM'} at 127.25.109.3:46993
I20260924 16:02:04.936759 26036 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 16:02:04.950063 28538 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:04.948486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43528 (local address 127.25.109.62:39397)
0924 16:02:04.948656 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:04.948662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:04.948883 (+ 221us) server_negotiation.cc:407] Connection header received
0924 16:02:04.949019 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:04.949021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:04.949062 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:04.949125 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:04.949918 (+ 793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43528: BlockingRecv error: recv got EOF from 127.0.0.1:43528 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 16:02:04.951082 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 28562
I20260924 16:02:04.956946 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 28699
I20260924 16:02:04.962442 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 28835
I20260924 16:02:04.967765 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 28466
2026-09-24T16:02:04Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2120 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:02:05 dist-test-slave-n106 krb5kdc[28977](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:02:05 dist-test-slave-n106 krb5kdc[28977](info): set up 2 sockets
Sep 24 16:02:05 dist-test-slave-n106 krb5kdc[28977](info): commencing operation
krb5kdc: starting...
W20260924 16:02:06.009933 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:02:06 dist-test-slave-n106 krb5kdc[28977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:02:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:02:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
I20260924 16:02:06.133391 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:40427
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:41877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:40427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:06.245208 28990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:06.245496 28990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:06.245589 28990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:06.249722 28990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:02:06.249804 28990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:06.249842 28990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:06.249900 28990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:02:06.249938 28990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:02:06.255154 28990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:41877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:40427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:40427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:06.256250 28990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:06.257383 28990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:06.266017 28998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:06.266084 28996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:06.266185 28995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:06.266268 28990 server_base.cc:1034] running on GCE node
I20260924 16:02:06.266896 28990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:06.267992 28990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:06.269191 28990 hybrid_clock.cc:648] HybridClock initialized: now 1790265726269157 us; error 43 us; skew 500 ppm
Sep 24 16:02:06 dist-test-slave-n106 krb5kdc[28977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265726, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:06.271927 28990 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:02:06.272921 28990 webserver.cc:460] Webserver started at http://127.25.109.62:41409/ using document root <none> and password file <none>
I20260924 16:02:06.273459 28990 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:06.273519 28990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:06.273691 28990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:06.275738 28990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "4cc1d06e527a456794d589be2d96ce99"
format_stamp: "Formatted at 2026-09-24 16:02:06 on dist-test-slave-n106"
server_key: "c68e86b513a23a1bebd5031a32bc9f6e"
server_key_iv: "adcd28e5479fe6523efaddd258c8d8de"
server_key_version: "encryptionkey@0"
I20260924 16:02:06.276249 28990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "4cc1d06e527a456794d589be2d96ce99"
format_stamp: "Formatted at 2026-09-24 16:02:06 on dist-test-slave-n106"
server_key: "c68e86b513a23a1bebd5031a32bc9f6e"
server_key_iv: "adcd28e5479fe6523efaddd258c8d8de"
server_key_version: "encryptionkey@0"
I20260924 16:02:06.280089 28990 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:02:06.282606 29005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:06.283862 28990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:02:06.283999 28990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "4cc1d06e527a456794d589be2d96ce99"
format_stamp: "Formatted at 2026-09-24 16:02:06 on dist-test-slave-n106"
server_key: "c68e86b513a23a1bebd5031a32bc9f6e"
server_key_iv: "adcd28e5479fe6523efaddd258c8d8de"
server_key_version: "encryptionkey@0"
I20260924 16:02:06.284113 28990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:06.304731 28990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:06.307682 28990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:06.307893 28990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:06.315178 28990 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:40427
I20260924 16:02:06.315481 29059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:40427 every 8 connection(s)
I20260924 16:02:06.316277 28990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:02:06.319157 29060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:06.319582 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 28990
I20260924 16:02:06.319717 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:02:06.320003 26036 external_mini_cluster.cc:1428] Setting key eca4ac9f39881031c1ff29301896b544
I20260924 16:02:06.326632 29060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cc1d06e527a456794d589be2d96ce99: Bootstrap starting.
I20260924 16:02:06.330165 29060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cc1d06e527a456794d589be2d96ce99: Neither blocks nor log segments found. Creating new log.
Sep 24 16:02:06 dist-test-slave-n106 krb5kdc[28977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:02:06.331226 29060 log.cc:824] T 00000000000000000000000000000000 P 4cc1d06e527a456794d589be2d96ce99: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:06.334519 29060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cc1d06e527a456794d589be2d96ce99: No bootstrap required, opened a new log
I20260924 16:02:06.336934 29063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:06.321436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56670 (local address 127.25.109.62:40427)
0924 16:02:06.321892 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:06.321912 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:06.321956 (+ 44us) server_negotiation.cc:407] Connection header received
0924 16:02:06.322720 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:06.322746 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:06.323184 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:06.323590 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:06.324305 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:06.325830 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:06.326707 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:06.327387 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:06.327549 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:06.331040 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:06.331047 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:06.331066 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:06.331107 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:06.333295 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:06.333860 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:06.333866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:06.333872 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:06.334018 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:06.334366 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:06.334369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:06.334371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:06.334932 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:06.335126 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:06.335466 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 16:02:06.335778 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":125,"threads_started":1}
I20260924 16:02:06.339093 29060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cc1d06e527a456794d589be2d96ce99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc1d06e527a456794d589be2d96ce99" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40427 } }
I20260924 16:02:06.339396 29060 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cc1d06e527a456794d589be2d96ce99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:06.339488 29060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cc1d06e527a456794d589be2d96ce99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc1d06e527a456794d589be2d96ce99, State: Initialized, Role: FOLLOWER
I20260924 16:02:06.339951 26036 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:02:06.340018 29060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cc1d06e527a456794d589be2d96ce99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc1d06e527a456794d589be2d96ce99" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40427 } }
I20260924 16:02:06.340200 29060 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cc1d06e527a456794d589be2d96ce99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:02:06.340271 29060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cc1d06e527a456794d589be2d96ce99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:02:06.340372 29060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cc1d06e527a456794d589be2d96ce99 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:06.341349 29060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cc1d06e527a456794d589be2d96ce99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc1d06e527a456794d589be2d96ce99" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40427 } }
I20260924 16:02:06.341729 29060 leader_election.cc:302] T 00000000000000000000000000000000 P 4cc1d06e527a456794d589be2d96ce99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cc1d06e527a456794d589be2d96ce99; no voters:
I20260924 16:02:06.342105 29060 leader_election.cc:288] T 00000000000000000000000000000000 P 4cc1d06e527a456794d589be2d96ce99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:02:06.342235 29066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cc1d06e527a456794d589be2d96ce99 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:06.342468 29066 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cc1d06e527a456794d589be2d96ce99 [term 1 LEADER]: Becoming Leader. State: Replica: 4cc1d06e527a456794d589be2d96ce99, State: Running, Role: LEADER
I20260924 16:02:06.342777 29066 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cc1d06e527a456794d589be2d96ce99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc1d06e527a456794d589be2d96ce99" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40427 } }
I20260924 16:02:06.343580 29060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cc1d06e527a456794d589be2d96ce99 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:02:06.343940 29066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cc1d06e527a456794d589be2d96ce99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cc1d06e527a456794d589be2d96ce99. Latest consensus state: current_term: 1 leader_uuid: "4cc1d06e527a456794d589be2d96ce99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc1d06e527a456794d589be2d96ce99" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40427 } } }
I20260924 16:02:06.344067 29066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cc1d06e527a456794d589be2d96ce99 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:06.344527 29070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:02:06.344540 29067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cc1d06e527a456794d589be2d96ce99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cc1d06e527a456794d589be2d96ce99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc1d06e527a456794d589be2d96ce99" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40427 } } }
I20260924 16:02:06.344755 29067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cc1d06e527a456794d589be2d96ce99 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:06.347859 29070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:02:06.353940 29070 catalog_manager.cc:1347] Generated new cluster ID: 4443db2e9515450987a14aaa4696ce9a
I20260924 16:02:06.354027 29070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:02:06.363488 29063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:06.357479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56686 (local address 127.25.109.62:40427)
0924 16:02:06.357606 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:06.357611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:06.357905 (+ 294us) server_negotiation.cc:407] Connection header received
0924 16:02:06.358048 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:06.358051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:06.358102 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:06.358181 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:06.358805 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:06.359570 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:06.360902 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:06.361304 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:06.361384 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:06.362448 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:06.362452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:06.362454 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:02:06.362996 (+ 542us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:06.363029 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:02:06.363048 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:06.363135 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:06.363269 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 16:02:06.363328 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 16:02:06.372294 29070 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:02:06.373504 29070 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:02:06.383400 29070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cc1d06e527a456794d589be2d96ce99: Generated new TSK 0
I20260924 16:02:06.384119 29070 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:02:06.389747 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 28990
2026-09-24T16:02:06Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1421 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:02:06 dist-test-slave-n106 krb5kdc[29093](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:02:06 dist-test-slave-n106 krb5kdc[29093](info): set up 2 sockets
Sep 24 16:02:06 dist-test-slave-n106 krb5kdc[29093](info): commencing operation
krb5kdc: starting...
W20260924 16:02:07.425482 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:02:07 dist-test-slave-n106 krb5kdc[29093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:02:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:02:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
I20260924 16:02:07.543320 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:38313
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:37219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:38313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:07.670083 29107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:07.670356 29107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:07.670395 29107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:07.674444 29107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:02:07.674506 29107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:07.674532 29107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:07.674549 29107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:02:07.674594 29107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:02:07.679880 29107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:37219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:38313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:38313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:07.680986 29107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:07.682026 29107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:02:07.690907 29107 server_base.cc:1034] running on GCE node
W20260924 16:02:07.691061 29115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:07.691053 29112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:07.691053 29113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:07.691767 29107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:07.692924 29107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:07.694113 29107 hybrid_clock.cc:648] HybridClock initialized: now 1790265727694092 us; error 32 us; skew 500 ppm
Sep 24 16:02:07 dist-test-slave-n106 krb5kdc[29093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265727, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:07.696791 29107 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:02:07.697683 29107 webserver.cc:460] Webserver started at http://127.25.109.62:45131/ using document root <none> and password file <none>
I20260924 16:02:07.698244 29107 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:07.698335 29107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:07.698572 29107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:07.700672 29107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "816465154d5f460a97ff32402cb579c7"
format_stamp: "Formatted at 2026-09-24 16:02:07 on dist-test-slave-n106"
server_key: "40541cb99dd0347ecfc6de7d45759d8e"
server_key_iv: "892eef7b245b8708fb0dceebee2c30c9"
server_key_version: "encryptionkey@0"
I20260924 16:02:07.701239 29107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "816465154d5f460a97ff32402cb579c7"
format_stamp: "Formatted at 2026-09-24 16:02:07 on dist-test-slave-n106"
server_key: "40541cb99dd0347ecfc6de7d45759d8e"
server_key_iv: "892eef7b245b8708fb0dceebee2c30c9"
server_key_version: "encryptionkey@0"
I20260924 16:02:07.704813 29107 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:02:07.707089 29122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:07.708230 29107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:02:07.708379 29107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "816465154d5f460a97ff32402cb579c7"
format_stamp: "Formatted at 2026-09-24 16:02:07 on dist-test-slave-n106"
server_key: "40541cb99dd0347ecfc6de7d45759d8e"
server_key_iv: "892eef7b245b8708fb0dceebee2c30c9"
server_key_version: "encryptionkey@0"
I20260924 16:02:07.708539 29107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:07.732805 29107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:07.735845 29107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:07.736071 29107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:07.743046 29107 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:38313
I20260924 16:02:07.743321 29175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:38313 every 8 connection(s)
I20260924 16:02:07.744108 29107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:02:07.746929 29176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:07.749559 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 29107
I20260924 16:02:07.749714 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:02:07.750025 26036 external_mini_cluster.cc:1428] Setting key 6a7e3693b7fa1e54e5ecf4576f5fb7a4
I20260924 16:02:07.753285 29176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 816465154d5f460a97ff32402cb579c7: Bootstrap starting.
I20260924 16:02:07.755540 29176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 816465154d5f460a97ff32402cb579c7: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:07.756280 29176 log.cc:824] T 00000000000000000000000000000000 P 816465154d5f460a97ff32402cb579c7: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:07.758245 29176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 816465154d5f460a97ff32402cb579c7: No bootstrap required, opened a new log
Sep 24 16:02:07 dist-test-slave-n106 krb5kdc[29093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:02:07.761480 29176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 816465154d5f460a97ff32402cb579c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816465154d5f460a97ff32402cb579c7" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 38313 } }
I20260924 16:02:07.761693 29176 raft_consensus.cc:374] T 00000000000000000000000000000000 P 816465154d5f460a97ff32402cb579c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:07.761751 29176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 816465154d5f460a97ff32402cb579c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816465154d5f460a97ff32402cb579c7, State: Initialized, Role: FOLLOWER
I20260924 16:02:07.762275 29176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 816465154d5f460a97ff32402cb579c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816465154d5f460a97ff32402cb579c7" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 38313 } }
I20260924 16:02:07.762485 29176 raft_consensus.cc:388] T 00000000000000000000000000000000 P 816465154d5f460a97ff32402cb579c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:02:07.762558 29176 raft_consensus.cc:482] T 00000000000000000000000000000000 P 816465154d5f460a97ff32402cb579c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:02:07.762686 29176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 816465154d5f460a97ff32402cb579c7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:07.763581 29176 raft_consensus.cc:504] T 00000000000000000000000000000000 P 816465154d5f460a97ff32402cb579c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816465154d5f460a97ff32402cb579c7" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 38313 } }
I20260924 16:02:07.763957 29176 leader_election.cc:302] T 00000000000000000000000000000000 P 816465154d5f460a97ff32402cb579c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 816465154d5f460a97ff32402cb579c7; no voters:
I20260924 16:02:07.764329 29176 leader_election.cc:288] T 00000000000000000000000000000000 P 816465154d5f460a97ff32402cb579c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:02:07.764459 29181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 816465154d5f460a97ff32402cb579c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:07.764667 29181 raft_consensus.cc:686] T 00000000000000000000000000000000 P 816465154d5f460a97ff32402cb579c7 [term 1 LEADER]: Becoming Leader. State: Replica: 816465154d5f460a97ff32402cb579c7, State: Running, Role: LEADER
I20260924 16:02:07.765026 29181 consensus_queue.cc:237] T 00000000000000000000000000000000 P 816465154d5f460a97ff32402cb579c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816465154d5f460a97ff32402cb579c7" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 38313 } }
I20260924 16:02:07.765235 29179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:07.751455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45918 (local address 127.25.109.62:38313)
0924 16:02:07.751900 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:07.751915 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:07.751950 (+ 35us) server_negotiation.cc:407] Connection header received
0924 16:02:07.752630 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:07.752647 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:07.752971 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:07.753322 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:07.754137 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:07.755137 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:07.756066 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:07.756622 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:07.756805 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:07.759472 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:07.759477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:07.759490 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:07.759517 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:07.761911 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:07.762377 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:07.762381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:07.762385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:07.762555 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:07.762873 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:07.762877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:07.762879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:07.763283 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:07.763504 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:07.763830 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 16:02:07.764168 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":133,"threads_started":1}
I20260924 16:02:07.765722 29176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 816465154d5f460a97ff32402cb579c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:02:07.766299 29181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 816465154d5f460a97ff32402cb579c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 816465154d5f460a97ff32402cb579c7. Latest consensus state: current_term: 1 leader_uuid: "816465154d5f460a97ff32402cb579c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816465154d5f460a97ff32402cb579c7" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 38313 } } }
I20260924 16:02:07.766425 29181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 816465154d5f460a97ff32402cb579c7 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:07.766870 29182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 816465154d5f460a97ff32402cb579c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "816465154d5f460a97ff32402cb579c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816465154d5f460a97ff32402cb579c7" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 38313 } } }
I20260924 16:02:07.767026 29182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 816465154d5f460a97ff32402cb579c7 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:07.767688 29187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:02:07.770574 29187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:02:07.775936 29187 catalog_manager.cc:1347] Generated new cluster ID: 5e5f833673154d21a55388b18f79ff13
I20260924 16:02:07.776000 29187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:02:07.796442 29187 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:02:07.801225 29187 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:02:07.805866 29187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 816465154d5f460a97ff32402cb579c7: Generated new TSK 0
I20260924 16:02:07.806663 29187 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
I20260924 16:02:07.839866 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:0
--local_ip_for_outbound_sockets=127.25.109.1
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:38313
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:37219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:07.950608 29202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:07.950876 29202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:07.950946 29202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:07.954967 29202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:07.955049 29202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:07.955144 29202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:02:07.960237 29202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:37219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:38313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:07.961277 29202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:07.962277 29202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:07.972038 29210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:07.972033 29208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:07.972268 29202 server_base.cc:1034] running on GCE node
W20260924 16:02:07.972033 29207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:07.972754 29202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:07.973402 29202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:07.974619 29202 hybrid_clock.cc:648] HybridClock initialized: now 1790265727974611 us; error 35 us; skew 500 ppm
Sep 24 16:02:07 dist-test-slave-n106 krb5kdc[29093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265727, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:07.987195 29202 init.cc:377] Logged in from keytab as kudu/127.25.109.1@KRBTEST.COM (short username kudu)
I20260924 16:02:07.988089 29202 webserver.cc:460] Webserver started at http://127.25.109.1:44797/ using document root <none> and password file <none>
I20260924 16:02:07.988724 29202 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:07.988787 29202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:07.989017 29202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:07.990994 29202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/data/instance:
uuid: "550b9769d7d64417aa0d4b9921333e04"
format_stamp: "Formatted at 2026-09-24 16:02:07 on dist-test-slave-n106"
server_key: "56d56f4e0ed6d520524cee144570ee9a"
server_key_iv: "9091b0f8cdf6c4915f4d2c9149b2bc54"
server_key_version: "encryptionkey@0"
I20260924 16:02:07.991554 29202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance:
uuid: "550b9769d7d64417aa0d4b9921333e04"
format_stamp: "Formatted at 2026-09-24 16:02:07 on dist-test-slave-n106"
server_key: "56d56f4e0ed6d520524cee144570ee9a"
server_key_iv: "9091b0f8cdf6c4915f4d2c9149b2bc54"
server_key_version: "encryptionkey@0"
I20260924 16:02:07.995239 29202 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 16:02:07.997722 29217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:07.998886 29202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:02:07.999007 29202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "550b9769d7d64417aa0d4b9921333e04"
format_stamp: "Formatted at 2026-09-24 16:02:07 on dist-test-slave-n106"
server_key: "56d56f4e0ed6d520524cee144570ee9a"
server_key_iv: "9091b0f8cdf6c4915f4d2c9149b2bc54"
server_key_version: "encryptionkey@0"
I20260924 16:02:07.999118 29202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:08.024945 29202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:08.027832 29202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:08.028046 29202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:08.028681 29202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:08.029523 29202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:08.029570 29202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:08.029639 29202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:08.029668 29202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:08.038363 29202 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:43961
I20260924 16:02:08.038388 29331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:43961 every 8 connection(s)
I20260924 16:02:08.039248 29202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:02:08.046743 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 29202
I20260924 16:02:08.046876 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance
I20260924 16:02:08.047209 26036 external_mini_cluster.cc:1428] Setting key 7cff456424fcff0a7866c43e6f5ac4b0
Sep 24 16:02:08 dist-test-slave-n106 krb5kdc[29093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265727, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:02:08.053777 29179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:08.041188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:33813 (local address 127.25.109.62:38313)
0924 16:02:08.041346 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:08.041350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:08.042199 (+ 849us) server_negotiation.cc:407] Connection header received
0924 16:02:08.043601 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:08.043603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:08.043650 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:08.043724 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:08.044903 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:08.045431 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:08.046691 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:08.047051 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:08.047164 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:08.049812 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:08.049819 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:08.049825 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:08.049859 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:08.051438 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:08.052077 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:08.052079 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:08.052080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:08.052155 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:08.052585 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:08.052587 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:08.052588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:08.052822 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:08.052925 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:08.053548 (+ 623us) server_negotiation.cc:299] Negotiation successful
0924 16:02:08.053657 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 16:02:08.055063 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:08.041554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:38313 (local address 127.25.109.1:33813)
0924 16:02:08.042045 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:08.042088 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:08.043381 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:08.043893 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:08.043899 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:08.044274 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:08.044765 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:08.044778 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:08.045562 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:08.045565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:08.046456 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:08.046467 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:08.047202 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:08.047206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:08.047405 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:08.048044 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:08.048060 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:08.049643 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:08.051559 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:08.051568 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:08.051587 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:08.051971 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:08.052244 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:08.052250 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:08.052255 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:08.052484 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:08.052940 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:08.052947 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:08.053277 (+ 330us) client_negotiation.cc:769] Sending connection context
0924 16:02:08.053498 (+ 221us) client_negotiation.cc:240] Negotiation successful
0924 16:02:08.053896 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":107,"threads_started":1}
I20260924 16:02:08.056368 29332 heartbeater.cc:347] Connected to a master server at 127.25.109.62:38313
I20260924 16:02:08.056648 29332 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.25.109.2@KRBTEST.COM; defaulting to no policy
I20260924 16:02:08.057179 29332 heartbeater.cc:511] Master 127.25.109.62:38313 requested a full tablet report, sending...
I20260924 16:02:08.058727 29139 ts_manager.cc:194] Registered new tserver with Master: 550b9769d7d64417aa0d4b9921333e04 (127.25.109.1:43961)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.2@KRBTEST.COM" created.
I20260924 16:02:08.060118 29139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.1@KRBTEST.COM'} at 127.25.109.1:33813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
I20260924 16:02:08.073439 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:0
--local_ip_for_outbound_sockets=127.25.109.2
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:38313
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:37219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:08.184921 29337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:08.185192 29337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:08.185276 29337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:08.189393 29337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:08.189452 29337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:08.189555 29337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:02:08.195632 29337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:37219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:38313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:08.196758 29337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:08.197796 29337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:08.207317 29343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:08.207453 29337 server_base.cc:1034] running on GCE node
W20260924 16:02:08.207329 29344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:08.207649 29346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:08.208026 29337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:08.208772 29337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:08.209969 29337 hybrid_clock.cc:648] HybridClock initialized: now 1790265728209953 us; error 29 us; skew 500 ppm
Sep 24 16:02:08 dist-test-slave-n106 krb5kdc[29093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265728, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:08.212853 29337 init.cc:377] Logged in from keytab as kudu/127.25.109.2@KRBTEST.COM (short username kudu)
I20260924 16:02:08.213757 29337 webserver.cc:460] Webserver started at http://127.25.109.2:40171/ using document root <none> and password file <none>
I20260924 16:02:08.214347 29337 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:08.214407 29337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:08.214651 29337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:08.216905 29337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/data/instance:
uuid: "60fa5c159a8941e3a79abb4957ee3277"
format_stamp: "Formatted at 2026-09-24 16:02:08 on dist-test-slave-n106"
server_key: "0668cce2a8d6be5d23cf78dcaddb4ff6"
server_key_iv: "f12484d6f934dc6fa1040039945e1668"
server_key_version: "encryptionkey@0"
I20260924 16:02:08.217530 29337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance:
uuid: "60fa5c159a8941e3a79abb4957ee3277"
format_stamp: "Formatted at 2026-09-24 16:02:08 on dist-test-slave-n106"
server_key: "0668cce2a8d6be5d23cf78dcaddb4ff6"
server_key_iv: "f12484d6f934dc6fa1040039945e1668"
server_key_version: "encryptionkey@0"
I20260924 16:02:08.221500 29337 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:02:08.223970 29353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:08.225261 29337 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:02:08.225384 29337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "60fa5c159a8941e3a79abb4957ee3277"
format_stamp: "Formatted at 2026-09-24 16:02:08 on dist-test-slave-n106"
server_key: "0668cce2a8d6be5d23cf78dcaddb4ff6"
server_key_iv: "f12484d6f934dc6fa1040039945e1668"
server_key_version: "encryptionkey@0"
I20260924 16:02:08.225492 29337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:08.256372 29337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:08.259867 29337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:08.260079 29337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:08.260799 29337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:08.261664 29337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:08.261710 29337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:08.261782 29337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:08.261823 29337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:08.271500 29337 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:46037
I20260924 16:02:08.271559 29469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:46037 every 8 connection(s)
I20260924 16:02:08.272469 29337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:02:08.279357 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 29337
I20260924 16:02:08.279466 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance
I20260924 16:02:08.279784 26036 external_mini_cluster.cc:1428] Setting key 2c42e6c882fc947709e552f687f165dc
Sep 24 16:02:08 dist-test-slave-n106 krb5kdc[29093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265728, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:02:08.286301 29179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:08.274254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:52355 (local address 127.25.109.62:38313)
0924 16:02:08.274399 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:08.274404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:08.275302 (+ 898us) server_negotiation.cc:407] Connection header received
0924 16:02:08.276643 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:08.276646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:08.276693 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:08.276768 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:08.277872 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:08.278384 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:08.279667 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:08.279911 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:08.280001 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:08.282722 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:08.282726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:08.282728 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:08.282760 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:08.284226 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:08.284774 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:08.284778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:08.284780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:08.284865 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:08.285221 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:08.285225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:08.285226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:08.285407 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:08.285519 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:08.286009 (+ 490us) server_negotiation.cc:299] Negotiation successful
0924 16:02:08.286133 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 16:02:08.287076 29472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:08.274716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:38313 (local address 127.25.109.2:52355)
0924 16:02:08.275154 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:08.275195 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:08.276380 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:08.276927 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:08.276933 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:08.277268 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:08.277745 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:08.277758 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:08.278488 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:08.278490 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:08.279554 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:08.279566 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:08.280065 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:08.280070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:08.280259 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:08.280938 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:08.280954 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:08.282554 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:08.284351 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:08.284357 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:08.284373 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:08.284667 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:08.284961 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:08.284965 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:08.284967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:08.285114 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:08.285516 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:08.285522 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:08.285765 (+ 243us) client_negotiation.cc:769] Sending connection context
0924 16:02:08.285949 (+ 184us) client_negotiation.cc:240] Negotiation successful
0924 16:02:08.286272 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":112,"threads_started":1}
I20260924 16:02:08.288306 29470 heartbeater.cc:347] Connected to a master server at 127.25.109.62:38313
I20260924 16:02:08.288570 29470 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.25.109.3@KRBTEST.COM; defaulting to no policy
I20260924 16:02:08.289088 29470 heartbeater.cc:511] Master 127.25.109.62:38313 requested a full tablet report, sending...
I20260924 16:02:08.290105 29139 ts_manager.cc:194] Registered new tserver with Master: 60fa5c159a8941e3a79abb4957ee3277 (127.25.109.2:46037)
I20260924 16:02:08.290674 29139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.2@KRBTEST.COM'} at 127.25.109.2:52355
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
I20260924 16:02:08.307734 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:0
--local_ip_for_outbound_sockets=127.25.109.3
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:38313
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:37219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:08.421331 29475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:08.421631 29475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:08.421705 29475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:08.426074 29475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:08.426149 29475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:08.426250 29475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:02:08.433490 29475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:37219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:38313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:08.434665 29475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:08.435746 29475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:08.445647 29483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:08.445703 29480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:08.445647 29481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:08.445756 29475 server_base.cc:1034] running on GCE node
I20260924 16:02:08.446292 29475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:08.446976 29475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:08.448167 29475 hybrid_clock.cc:648] HybridClock initialized: now 1790265728448150 us; error 27 us; skew 500 ppm
Sep 24 16:02:08 dist-test-slave-n106 krb5kdc[29093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265728, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:08.451048 29475 init.cc:377] Logged in from keytab as kudu/127.25.109.3@KRBTEST.COM (short username kudu)
I20260924 16:02:08.451908 29475 webserver.cc:460] Webserver started at http://127.25.109.3:43659/ using document root <none> and password file <none>
I20260924 16:02:08.452498 29475 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:08.452566 29475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:08.452749 29475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:08.454741 29475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/data/instance:
uuid: "3bddd37daf244330bdec06bd2aac4545"
format_stamp: "Formatted at 2026-09-24 16:02:08 on dist-test-slave-n106"
server_key: "4df329380358bf3aa5ddc2fe810a10d2"
server_key_iv: "4cfee6cc285cca3e3d9f9f86f712dfa9"
server_key_version: "encryptionkey@0"
I20260924 16:02:08.455268 29475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance:
uuid: "3bddd37daf244330bdec06bd2aac4545"
format_stamp: "Formatted at 2026-09-24 16:02:08 on dist-test-slave-n106"
server_key: "4df329380358bf3aa5ddc2fe810a10d2"
server_key_iv: "4cfee6cc285cca3e3d9f9f86f712dfa9"
server_key_version: "encryptionkey@0"
I20260924 16:02:08.459160 29475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 16:02:08.461736 29490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:08.463052 29475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:02:08.463158 29475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "3bddd37daf244330bdec06bd2aac4545"
format_stamp: "Formatted at 2026-09-24 16:02:08 on dist-test-slave-n106"
server_key: "4df329380358bf3aa5ddc2fe810a10d2"
server_key_iv: "4cfee6cc285cca3e3d9f9f86f712dfa9"
server_key_version: "encryptionkey@0"
I20260924 16:02:08.463254 29475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:08.489981 29475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:08.493567 29475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:08.493747 29475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:08.494339 29475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:08.495189 29475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:08.495237 29475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:08.495278 29475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:08.495294 29475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:08.504915 29475 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:43201
I20260924 16:02:08.504993 29605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:43201 every 8 connection(s)
I20260924 16:02:08.505869 29475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:02:08.514559 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 29475
I20260924 16:02:08.514717 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance
I20260924 16:02:08.515018 26036 external_mini_cluster.cc:1428] Setting key 67d90312297295108ff7e8d4ab203af8
Sep 24 16:02:08 dist-test-slave-n106 krb5kdc[29093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265728, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:02:08.519941 29179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:08.507713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:54697 (local address 127.25.109.62:38313)
0924 16:02:08.507866 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:08.507871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:08.508817 (+ 946us) server_negotiation.cc:407] Connection header received
0924 16:02:08.510165 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:08.510167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:08.510215 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:08.510290 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:08.511467 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:08.512041 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:08.513388 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:08.513598 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:08.513681 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:08.516646 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:08.516649 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:08.516651 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:08.516674 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:08.518187 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:08.518657 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:08.518660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:08.518661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:08.518735 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:08.519028 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:08.519030 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:08.519031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:08.519199 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:08.519272 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:08.519730 (+ 458us) server_negotiation.cc:299] Negotiation successful
0924 16:02:08.519824 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 16:02:08.520852 29608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:08.508158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:38313 (local address 127.25.109.3:54697)
0924 16:02:08.508670 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:08.508713 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:08.509959 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:08.510489 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:08.510496 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:08.510852 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:08.511323 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:08.511336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:08.512149 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:08.512152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:08.513273 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:08.513283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:08.513737 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:08.513740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:08.513886 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:08.514463 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:08.514479 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:08.516281 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:08.518290 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:08.518296 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:08.518308 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:08.518568 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:08.518813 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:08.518815 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:08.518817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:08.518950 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:08.519298 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:08.519303 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:08.519529 (+ 226us) client_negotiation.cc:769] Sending connection context
0924 16:02:08.519684 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 16:02:08.519932 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":116,"threads_started":1}
I20260924 16:02:08.521998 29606 heartbeater.cc:347] Connected to a master server at 127.25.109.62:38313
I20260924 16:02:08.522257 29606 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:08.522754 29606 heartbeater.cc:511] Master 127.25.109.62:38313 requested a full tablet report, sending...
I20260924 16:02:08.523825 29139 ts_manager.cc:194] Registered new tserver with Master: 3bddd37daf244330bdec06bd2aac4545 (127.25.109.3:43201)
I20260924 16:02:08.524394 29139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.3@KRBTEST.COM'} at 127.25.109.3:54697
I20260924 16:02:08.529093 26036 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 16:02:08 dist-test-slave-n106 krb5kdc[29093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265728, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 16:02:08 dist-test-slave-n106 krb5kdc[29093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265728, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:02:08.554495 29179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:08.546239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45920 (local address 127.25.109.62:38313)
0924 16:02:08.546395 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:08.546398 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:08.546549 (+ 151us) server_negotiation.cc:407] Connection header received
0924 16:02:08.546674 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:08.546677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:08.546723 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:08.546798 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:08.547490 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:08.548022 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:08.548886 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:08.549094 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:08.549172 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:08.551566 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:08.551570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:08.551572 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:08.551596 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:08.553095 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:08.553459 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:08.553462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:08.553463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:08.553538 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:08.553761 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:08.553763 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:08.553764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:08.553979 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:08.554080 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:08.554275 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 16:02:08.554371 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260924 16:02:08.554926 29139 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45920
I20260924 16:02:08.555792 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 29202
I20260924 16:02:08.561822 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 29337
I20260924 16:02:08.568225 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 29475
I20260924 16:02:08.573851 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 29107
2026-09-24T16:02:08Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2185 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:02:08 dist-test-slave-n106 krb5kdc[29618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:02:08 dist-test-slave-n106 krb5kdc[29618](info): set up 2 sockets
Sep 24 16:02:08 dist-test-slave-n106 krb5kdc[29618](info): commencing operation
krb5kdc: starting...
W20260924 16:02:09.613440 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:02:09 dist-test-slave-n106 krb5kdc[29618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:02:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:02:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
I20260924 16:02:09.730949 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:37279
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:38617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:37279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:09.842083 29631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:09.842351 29631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:09.842392 29631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:09.846481 29631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:02:09.846539 29631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:09.846558 29631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:09.846613 29631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:02:09.846645 29631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:02:09.851863 29631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:38617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:37279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:37279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:09.852990 29631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:09.854056 29631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:02:09.862665 29631 server_base.cc:1034] running on GCE node
W20260924 16:02:09.862625 29637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:09.862591 29639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:09.862609 29636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:09.863441 29631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:09.864573 29631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:09.865748 29631 hybrid_clock.cc:648] HybridClock initialized: now 1790265729865727 us; error 31 us; skew 500 ppm
Sep 24 16:02:09 dist-test-slave-n106 krb5kdc[29618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265729, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:09.868604 29631 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:02:09.869555 29631 webserver.cc:460] Webserver started at http://127.25.109.62:32897/ using document root <none> and password file <none>
I20260924 16:02:09.870146 29631 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:09.870236 29631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:09.870476 29631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:09.872638 29631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "5032120a6f124d829cb98b291ad22e49"
format_stamp: "Formatted at 2026-09-24 16:02:09 on dist-test-slave-n106"
server_key: "0a3faba6f1dcd6e2d5270e7c831981b4"
server_key_iv: "a22e2ff76cf26dc613b8370dfafd5eac"
server_key_version: "encryptionkey@0"
I20260924 16:02:09.873241 29631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "5032120a6f124d829cb98b291ad22e49"
format_stamp: "Formatted at 2026-09-24 16:02:09 on dist-test-slave-n106"
server_key: "0a3faba6f1dcd6e2d5270e7c831981b4"
server_key_iv: "a22e2ff76cf26dc613b8370dfafd5eac"
server_key_version: "encryptionkey@0"
I20260924 16:02:09.877156 29631 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:02:09.879711 29646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:09.881058 29631 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:02:09.881217 29631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "5032120a6f124d829cb98b291ad22e49"
format_stamp: "Formatted at 2026-09-24 16:02:09 on dist-test-slave-n106"
server_key: "0a3faba6f1dcd6e2d5270e7c831981b4"
server_key_iv: "a22e2ff76cf26dc613b8370dfafd5eac"
server_key_version: "encryptionkey@0"
I20260924 16:02:09.881336 29631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:09.897125 29631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:09.900465 29631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:09.900674 29631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:09.907655 29631 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:37279
I20260924 16:02:09.907727 29699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:37279 every 8 connection(s)
I20260924 16:02:09.908655 29631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:02:09.911418 29700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:09.916761 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 29631
I20260924 16:02:09.916917 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:02:09.917074 29700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5032120a6f124d829cb98b291ad22e49: Bootstrap starting.
I20260924 16:02:09.917225 26036 external_mini_cluster.cc:1428] Setting key 2015818cdbf6fcc8ff0d2456a933ab9e
I20260924 16:02:09.919612 29700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5032120a6f124d829cb98b291ad22e49: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:09.920362 29700 log.cc:824] T 00000000000000000000000000000000 P 5032120a6f124d829cb98b291ad22e49: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:09.922261 29700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5032120a6f124d829cb98b291ad22e49: No bootstrap required, opened a new log
Sep 24 16:02:09 dist-test-slave-n106 krb5kdc[29618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:02:09.926083 29700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5032120a6f124d829cb98b291ad22e49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5032120a6f124d829cb98b291ad22e49" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 37279 } }
I20260924 16:02:09.926375 29700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5032120a6f124d829cb98b291ad22e49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:09.926446 29700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5032120a6f124d829cb98b291ad22e49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5032120a6f124d829cb98b291ad22e49, State: Initialized, Role: FOLLOWER
I20260924 16:02:09.926980 29700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5032120a6f124d829cb98b291ad22e49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5032120a6f124d829cb98b291ad22e49" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 37279 } }
I20260924 16:02:09.927141 29700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5032120a6f124d829cb98b291ad22e49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:02:09.927223 29700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5032120a6f124d829cb98b291ad22e49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:02:09.927338 29700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5032120a6f124d829cb98b291ad22e49 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:09.928582 29700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5032120a6f124d829cb98b291ad22e49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5032120a6f124d829cb98b291ad22e49" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 37279 } }
I20260924 16:02:09.928963 29700 leader_election.cc:302] T 00000000000000000000000000000000 P 5032120a6f124d829cb98b291ad22e49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5032120a6f124d829cb98b291ad22e49; no voters:
I20260924 16:02:09.929327 29700 leader_election.cc:288] T 00000000000000000000000000000000 P 5032120a6f124d829cb98b291ad22e49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:02:09.929397 29705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5032120a6f124d829cb98b291ad22e49 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:09.929600 29705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5032120a6f124d829cb98b291ad22e49 [term 1 LEADER]: Becoming Leader. State: Replica: 5032120a6f124d829cb98b291ad22e49, State: Running, Role: LEADER
I20260924 16:02:09.929955 29705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5032120a6f124d829cb98b291ad22e49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5032120a6f124d829cb98b291ad22e49" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 37279 } }
I20260924 16:02:09.930665 29700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5032120a6f124d829cb98b291ad22e49 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:02:09.931133 29705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5032120a6f124d829cb98b291ad22e49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5032120a6f124d829cb98b291ad22e49. Latest consensus state: current_term: 1 leader_uuid: "5032120a6f124d829cb98b291ad22e49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5032120a6f124d829cb98b291ad22e49" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 37279 } } }
I20260924 16:02:09.931321 29705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5032120a6f124d829cb98b291ad22e49 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:09.931532 29706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5032120a6f124d829cb98b291ad22e49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5032120a6f124d829cb98b291ad22e49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5032120a6f124d829cb98b291ad22e49" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 37279 } } }
I20260924 16:02:09.931777 29710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:02:09.932291 29703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:09.918562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49264 (local address 127.25.109.62:37279)
0924 16:02:09.918981 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:09.918996 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:09.919031 (+ 35us) server_negotiation.cc:407] Connection header received
0924 16:02:09.919652 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:09.919669 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:09.920054 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:09.920406 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:09.921277 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:09.922270 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:09.923152 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:09.923659 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:09.923851 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:09.926564 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:09.926571 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:09.926595 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:09.926625 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:09.928745 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:09.929226 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:09.929233 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:09.929239 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:09.929367 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:09.929747 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:09.929750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:09.929752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:09.930281 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:09.930497 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:09.930828 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 16:02:09.931314 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":124,"threads_started":1}
I20260924 16:02:09.931718 29706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5032120a6f124d829cb98b291ad22e49 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:09.936373 29710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:02:09.942408 29710 catalog_manager.cc:1347] Generated new cluster ID: 56698b521c6f45beb54a9ae242bd0f0f
I20260924 16:02:09.942487 29710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:02:09.960554 29710 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:02:09.961895 29710 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:02:09.969901 29710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5032120a6f124d829cb98b291ad22e49: Generated new TSK 0
I20260924 16:02:09.970743 29710 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
I20260924 16:02:09.997684 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:0
--local_ip_for_outbound_sockets=127.25.109.1
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:37279
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:38617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:10.110364 29726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:10.110612 29726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:10.110654 29726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:10.114789 29726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:10.114841 29726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:10.114926 29726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:02:10.120056 29726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:38617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:37279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:10.121198 29726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:10.122246 29726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:10.131892 29731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:10.131892 29732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:10.131914 29734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:10.132088 29726 server_base.cc:1034] running on GCE node
I20260924 16:02:10.132674 29726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:10.133332 29726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:10.134505 29726 hybrid_clock.cc:648] HybridClock initialized: now 1790265730134490 us; error 43 us; skew 500 ppm
Sep 24 16:02:10 dist-test-slave-n106 krb5kdc[29618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265730, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:10.137382 29726 init.cc:377] Logged in from keytab as kudu/127.25.109.1@KRBTEST.COM (short username kudu)
I20260924 16:02:10.138294 29726 webserver.cc:460] Webserver started at http://127.25.109.1:44091/ using document root <none> and password file <none>
I20260924 16:02:10.138862 29726 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:10.138953 29726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:10.139196 29726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:10.141321 29726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-0/data/instance:
uuid: "cb4bcfcf564c4c548dc348eacc021a4b"
format_stamp: "Formatted at 2026-09-24 16:02:10 on dist-test-slave-n106"
server_key: "d491913dd558968c8d65d632b594b02a"
server_key_iv: "3521c4ec13cf15a7a9f717086b7d918b"
server_key_version: "encryptionkey@0"
I20260924 16:02:10.141901 29726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance:
uuid: "cb4bcfcf564c4c548dc348eacc021a4b"
format_stamp: "Formatted at 2026-09-24 16:02:10 on dist-test-slave-n106"
server_key: "d491913dd558968c8d65d632b594b02a"
server_key_iv: "3521c4ec13cf15a7a9f717086b7d918b"
server_key_version: "encryptionkey@0"
I20260924 16:02:10.145668 29726 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:02:10.148079 29741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:10.149319 29726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:02:10.149459 29726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "cb4bcfcf564c4c548dc348eacc021a4b"
format_stamp: "Formatted at 2026-09-24 16:02:10 on dist-test-slave-n106"
server_key: "d491913dd558968c8d65d632b594b02a"
server_key_iv: "3521c4ec13cf15a7a9f717086b7d918b"
server_key_version: "encryptionkey@0"
I20260924 16:02:10.149573 29726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:10.167466 29726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:10.170912 29726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:10.171133 29726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:10.171742 29726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:10.172652 29726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:10.172721 29726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:10.172794 29726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:10.172847 29726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:10.182453 29726 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:44625
I20260924 16:02:10.182473 29856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:44625 every 8 connection(s)
I20260924 16:02:10.183358 29726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:02:10.184310 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 29726
I20260924 16:02:10.184481 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance
I20260924 16:02:10.184832 26036 external_mini_cluster.cc:1428] Setting key febbbb17ff72bca6a74ffc189fbe9a00
WARNING: no policy specified for kudu/127.25.109.2@KRBTEST.COM; defaulting to no policy
Sep 24 16:02:10 dist-test-slave-n106 krb5kdc[29618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265730, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.2@KRBTEST.COM" created.
I20260924 16:02:10.199409 29703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:10.185460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:35449 (local address 127.25.109.62:37279)
0924 16:02:10.185612 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:10.185615 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:10.186650 (+ 1035us) server_negotiation.cc:407] Connection header received
0924 16:02:10.188284 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:10.188287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:10.188334 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:10.188447 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:10.190087 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:10.190694 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:10.192018 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:10.192480 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:10.192599 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:10.195112 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:10.195115 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:10.195117 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:10.195142 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:10.196976 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:10.197641 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:10.197643 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:10.197644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:10.197721 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:10.198122 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:10.198124 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:10.198125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:10.198304 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:10.198378 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:10.199023 (+ 645us) server_negotiation.cc:299] Negotiation successful
0924 16:02:10.199176 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 16:02:10.200204 29859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:10.185963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:37279 (local address 127.25.109.1:35449)
0924 16:02:10.186513 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:10.186556 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:10.188038 (+ 1482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:10.188654 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:10.188664 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:10.189215 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:10.189916 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:10.189939 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:10.190833 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:10.190839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:10.191878 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:10.191888 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:10.192680 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:10.192684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:10.192835 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:10.193479 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:10.193497 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:10.194943 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:10.197100 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:10.197107 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:10.197139 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:10.197529 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:10.197815 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:10.197821 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:10.197826 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:10.198026 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:10.198409 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:10.198418 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:10.198765 (+ 347us) client_negotiation.cc:769] Sending connection context
0924 16:02:10.198935 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 16:02:10.199235 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":130,"threads_started":1}
I20260924 16:02:10.201581 29857 heartbeater.cc:347] Connected to a master server at 127.25.109.62:37279
I20260924 16:02:10.201843 29857 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:10.202364 29857 heartbeater.cc:511] Master 127.25.109.62:37279 requested a full tablet report, sending...
I20260924 16:02:10.203960 29664 ts_manager.cc:194] Registered new tserver with Master: cb4bcfcf564c4c548dc348eacc021a4b (127.25.109.1:44625)
I20260924 16:02:10.205184 29664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.1@KRBTEST.COM'} at 127.25.109.1:35449
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
I20260924 16:02:10.213559 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:0
--local_ip_for_outbound_sockets=127.25.109.2
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:37279
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:38617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:10.327291 29863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:10.327558 29863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:10.327601 29863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:10.331836 29863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:10.331964 29863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:10.332082 29863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:02:10.340210 29863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:38617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:37279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:10.341359 29863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:10.342449 29863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:10.352200 29871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:10.352223 29869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:10.352378 29863 server_base.cc:1034] running on GCE node
W20260924 16:02:10.352223 29868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:10.353031 29863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:10.353706 29863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:10.354895 29863 hybrid_clock.cc:648] HybridClock initialized: now 1790265730354872 us; error 33 us; skew 500 ppm
Sep 24 16:02:10 dist-test-slave-n106 krb5kdc[29618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265730, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:10.357641 29863 init.cc:377] Logged in from keytab as kudu/127.25.109.2@KRBTEST.COM (short username kudu)
I20260924 16:02:10.358507 29863 webserver.cc:460] Webserver started at http://127.25.109.2:42447/ using document root <none> and password file <none>
I20260924 16:02:10.359084 29863 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:10.359189 29863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:10.359426 29863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:10.361545 29863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-1/data/instance:
uuid: "6549efb81f15438dad173234a5611db4"
format_stamp: "Formatted at 2026-09-24 16:02:10 on dist-test-slave-n106"
server_key: "50b0161bec75363699a2040569fa4b3f"
server_key_iv: "330f7f3edfa3dab8451b879b773a0ff2"
server_key_version: "encryptionkey@0"
I20260924 16:02:10.362243 29863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance:
uuid: "6549efb81f15438dad173234a5611db4"
format_stamp: "Formatted at 2026-09-24 16:02:10 on dist-test-slave-n106"
server_key: "50b0161bec75363699a2040569fa4b3f"
server_key_iv: "330f7f3edfa3dab8451b879b773a0ff2"
server_key_version: "encryptionkey@0"
I20260924 16:02:10.366864 29863 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:02:10.369464 29878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:10.370826 29863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:02:10.370955 29863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "6549efb81f15438dad173234a5611db4"
format_stamp: "Formatted at 2026-09-24 16:02:10 on dist-test-slave-n106"
server_key: "50b0161bec75363699a2040569fa4b3f"
server_key_iv: "330f7f3edfa3dab8451b879b773a0ff2"
server_key_version: "encryptionkey@0"
I20260924 16:02:10.371059 29863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:10.399786 29863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:10.402801 29863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:10.402953 29863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:10.403563 29863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:10.404460 29863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:10.404518 29863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:10.404578 29863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:10.404610 29863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:10.413269 29863 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:32895
I20260924 16:02:10.413297 29992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:32895 every 8 connection(s)
I20260924 16:02:10.414145 29863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:02:10.419960 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 29863
I20260924 16:02:10.420133 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance
I20260924 16:02:10.420503 26036 external_mini_cluster.cc:1428] Setting key 7a9a3c31c65f1c1cb3882e2f43d06115
Sep 24 16:02:10 dist-test-slave-n106 krb5kdc[29618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265730, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:02:10.430583 29703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:10.415838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:55149 (local address 127.25.109.62:37279)
0924 16:02:10.415971 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:10.415975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:10.416835 (+ 860us) server_negotiation.cc:407] Connection header received
0924 16:02:10.418259 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:10.418263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:10.418329 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:10.418434 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:10.419588 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:10.420196 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:10.422002 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:10.422324 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:10.422409 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:10.426109 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:10.426116 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:10.426121 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:10.426152 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:10.428079 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:10.428766 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:10.428769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:10.428770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:10.428848 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:10.429314 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:10.429316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:10.429317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:10.429500 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:10.429575 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:10.430327 (+ 752us) server_negotiation.cc:299] Negotiation successful
0924 16:02:10.430457 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 16:02:10.432096 29996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:10.416239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:37279 (local address 127.25.109.2:55149)
0924 16:02:10.416686 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:10.416729 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:10.418036 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:10.418626 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:10.418633 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:10.418976 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:10.419439 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:10.419452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:10.420387 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:10.420393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:10.421863 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:10.421874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:10.422630 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:10.422634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:10.422839 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:10.423555 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:10.423614 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:10.425905 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:10.428210 (+ 2305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:10.428219 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:10.428239 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:10.428659 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:10.428940 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:10.428944 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:10.428948 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:10.429194 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:10.429611 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:10.429620 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:10.429996 (+ 376us) client_negotiation.cc:769] Sending connection context
0924 16:02:10.430251 (+ 255us) client_negotiation.cc:240] Negotiation successful
0924 16:02:10.430634 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.25.109.3@KRBTEST.COM; defaulting to no policy
I20260924 16:02:10.433390 29993 heartbeater.cc:347] Connected to a master server at 127.25.109.62:37279
I20260924 16:02:10.433676 29993 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:10.434176 29993 heartbeater.cc:511] Master 127.25.109.62:37279 requested a full tablet report, sending...
I20260924 16:02:10.435797 29664 ts_manager.cc:194] Registered new tserver with Master: 6549efb81f15438dad173234a5611db4 (127.25.109.2:32895)
I20260924 16:02:10.436384 29664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.2@KRBTEST.COM'} at 127.25.109.2:55149
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.3@KRBTEST.COM" created.
I20260924 16:02:11.207706 29857 heartbeater.cc:503] Master 127.25.109.62:37279 was elected leader, sending a full tablet report...
I20260924 16:02:11.438457 29993 heartbeater.cc:503] Master 127.25.109.62:37279 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
W20260924 16:02:11.829921 26036 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.25.109.3: real 1.407s user 0.000s sys 0.004s
I20260924 16:02:11.830488 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:0
--local_ip_for_outbound_sockets=127.25.109.3
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:37279
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:38617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:11.950275 29999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:11.950551 29999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:11.950600 29999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:11.955200 29999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:11.955262 29999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:11.955322 29999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:02:11.960951 29999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:38617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:37279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:11.962021 29999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:11.963092 29999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:11.973390 30004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:11.973383 30005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:11.973387 30007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:11.973721 29999 server_base.cc:1034] running on GCE node
I20260924 16:02:11.974269 29999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:11.974951 29999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:11.976125 29999 hybrid_clock.cc:648] HybridClock initialized: now 1790265731976105 us; error 31 us; skew 500 ppm
Sep 24 16:02:11 dist-test-slave-n106 krb5kdc[29618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265731, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:11.978909 29999 init.cc:377] Logged in from keytab as kudu/127.25.109.3@KRBTEST.COM (short username kudu)
I20260924 16:02:11.979756 29999 webserver.cc:460] Webserver started at http://127.25.109.3:42101/ using document root <none> and password file <none>
I20260924 16:02:11.980310 29999 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:11.980368 29999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:11.980613 29999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:11.982684 29999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-2/data/instance:
uuid: "c03eb3cb54cb4f78bfb135300d3019ef"
format_stamp: "Formatted at 2026-09-24 16:02:11 on dist-test-slave-n106"
server_key: "a0c2d9895f42e20708117887b6eb4fcc"
server_key_iv: "08eaabecace14a1a7a40e4618542df43"
server_key_version: "encryptionkey@0"
I20260924 16:02:11.983227 29999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance:
uuid: "c03eb3cb54cb4f78bfb135300d3019ef"
format_stamp: "Formatted at 2026-09-24 16:02:11 on dist-test-slave-n106"
server_key: "a0c2d9895f42e20708117887b6eb4fcc"
server_key_iv: "08eaabecace14a1a7a40e4618542df43"
server_key_version: "encryptionkey@0"
I20260924 16:02:11.987401 29999 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 16:02:11.989832 30014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:11.991091 29999 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:02:11.991240 29999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "c03eb3cb54cb4f78bfb135300d3019ef"
format_stamp: "Formatted at 2026-09-24 16:02:11 on dist-test-slave-n106"
server_key: "a0c2d9895f42e20708117887b6eb4fcc"
server_key_iv: "08eaabecace14a1a7a40e4618542df43"
server_key_version: "encryptionkey@0"
I20260924 16:02:11.991353 29999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:12.005017 29999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:12.008176 29999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:12.008448 29999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:12.009112 29999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:12.009980 29999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:12.010053 29999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:12.010129 29999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:12.010170 29999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:12.020668 29999 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:36181
I20260924 16:02:12.020686 30129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:36181 every 8 connection(s)
I20260924 16:02:12.021695 29999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:02:12.027534 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 29999
I20260924 16:02:12.027670 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance
I20260924 16:02:12.027971 26036 external_mini_cluster.cc:1428] Setting key 8ae8f3a37568c82d223b52ad9cc165e6
Sep 24 16:02:12 dist-test-slave-n106 krb5kdc[29618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265731, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:02:12.037092 30132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:12.023739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:50919 (local address 127.25.109.62:37279)
0924 16:02:12.024029 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:12.024033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:12.024781 (+ 748us) server_negotiation.cc:407] Connection header received
0924 16:02:12.026188 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:12.026191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:12.026254 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:12.026372 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:12.027831 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:12.028570 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:12.030075 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:12.030374 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:12.030461 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:12.033323 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:12.033328 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:12.033330 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:12.033364 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:12.034753 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:12.035307 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:12.035311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:12.035313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:12.035404 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:12.035774 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:12.035779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:12.035780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:12.035996 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:12.036111 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:12.036743 (+ 632us) server_negotiation.cc:299] Negotiation successful
0924 16:02:12.036884 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":111,"threads_started":1}
I20260924 16:02:12.037954 30133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:12.024119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:37279 (local address 127.25.109.3:50919)
0924 16:02:12.024633 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:12.024678 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:12.025973 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:12.026600 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:12.026607 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:12.026967 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:12.027641 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:12.027666 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:12.028704 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:12.028709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:12.029920 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:12.029930 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:12.030548 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:12.030552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:12.030719 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:12.031378 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:12.031395 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:12.033137 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:12.034882 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:12.034889 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:12.034906 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:12.035186 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:12.035506 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:12.035509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:12.035511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:12.035668 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:12.036111 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:12.036120 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:12.036394 (+ 274us) client_negotiation.cc:769] Sending connection context
0924 16:02:12.036689 (+ 295us) client_negotiation.cc:240] Negotiation successful
0924 16:02:12.037052 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":103,"threads_started":1}
I20260924 16:02:12.039259 30130 heartbeater.cc:347] Connected to a master server at 127.25.109.62:37279
I20260924 16:02:12.039541 30130 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:12.040071 30130 heartbeater.cc:511] Master 127.25.109.62:37279 requested a full tablet report, sending...
I20260924 16:02:12.041286 29663 ts_manager.cc:194] Registered new tserver with Master: c03eb3cb54cb4f78bfb135300d3019ef (127.25.109.3:36181)
I20260924 16:02:12.042114 26036 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:02:12.042114 29663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.3@KRBTEST.COM'} at 127.25.109.3:50919
Sep 24 16:02:12 dist-test-slave-n106 krb5kdc[29618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 16:02:12 dist-test-slave-n106 krb5kdc[29618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.1@KRBTEST.COM
I20260924 16:02:12.130055 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:12.119912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57250 (local address 127.25.109.1:44625)
0924 16:02:12.120303 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:12.120311 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:12.120338 (+ 27us) server_negotiation.cc:407] Connection header received
0924 16:02:12.120394 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:12.120421 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:12.120641 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:12.120776 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:12.121593 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:12.122386 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:12.123307 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:12.123716 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:12.123799 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:12.126477 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:12.126487 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:12.126502 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:12.126528 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:12.128153 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:12.128668 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:12.128675 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:12.128676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:12.128765 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:12.129021 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:12.129024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:12.129025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:12.129287 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:12.129463 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:12.129652 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 16:02:12.129836 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":85,"threads_started":1}
Sep 24 16:02:12 dist-test-slave-n106 krb5kdc[29618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:02:12.142481 30132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:12.133525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49272 (local address 127.25.109.62:37279)
0924 16:02:12.133689 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:12.133695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:12.133788 (+ 93us) server_negotiation.cc:407] Connection header received
0924 16:02:12.133975 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:12.133978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:12.134035 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:12.134146 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:12.134810 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:12.135362 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:12.136179 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:12.136508 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:12.136625 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:12.139273 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:12.139277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:12.139280 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:12.139319 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:12.140672 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:12.141122 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:12.141126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:12.141128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:12.141220 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:12.141508 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:12.141511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:12.141513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:12.141764 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:12.141911 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:12.142162 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 16:02:12.142301 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260924 16:02:12.142984 29663 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49272
I20260924 16:02:12.144124 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 29726
I20260924 16:02:12.150826 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 29863
I20260924 16:02:12.156654 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 29999
I20260924 16:02:12.162214 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 29631
2026-09-24T16:02:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3589 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:02:12 dist-test-slave-n106 krb5kdc[30144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:02:12 dist-test-slave-n106 krb5kdc[30144](info): set up 2 sockets
Sep 24 16:02:12 dist-test-slave-n106 krb5kdc[30144](info): commencing operation
krb5kdc: starting...
W20260924 16:02:13.201498 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:02:13 dist-test-slave-n106 krb5kdc[30144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:02:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:02:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
I20260924 16:02:13.321709 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:40385
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:33829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:40385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:13.437304 30157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:13.437626 30157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:13.437723 30157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:13.441830 30157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:02:13.441913 30157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:13.441951 30157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:13.442008 30157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:02:13.442045 30157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:02:13.447446 30157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:33829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:40385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:40385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:13.448640 30157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:13.449687 30157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:13.457953 30165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:13.457959 30163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:13.458122 30157 server_base.cc:1034] running on GCE node
W20260924 16:02:13.458014 30162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:13.458916 30157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:13.459983 30157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:13.461166 30157 hybrid_clock.cc:648] HybridClock initialized: now 1790265733461152 us; error 25 us; skew 500 ppm
Sep 24 16:02:13 dist-test-slave-n106 krb5kdc[30144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265733, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:13.468204 30157 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:02:13.469079 30157 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:13.469177 30157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:13.469414 30157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:13.471529 30157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "6dfd51f8a1134e67b2e40461f6b37041"
format_stamp: "Formatted at 2026-09-24 16:02:13 on dist-test-slave-n106"
server_key: "dab7c2613f43b5ffa7cca5ae79fc6108"
server_key_iv: "8c6037ec1a1cef64805815e82b601a6a"
server_key_version: "encryptionkey@0"
I20260924 16:02:13.472134 30157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "6dfd51f8a1134e67b2e40461f6b37041"
format_stamp: "Formatted at 2026-09-24 16:02:13 on dist-test-slave-n106"
server_key: "dab7c2613f43b5ffa7cca5ae79fc6108"
server_key_iv: "8c6037ec1a1cef64805815e82b601a6a"
server_key_version: "encryptionkey@0"
I20260924 16:02:13.475934 30157 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:02:13.478510 30171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:13.479745 30157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:02:13.479899 30157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "6dfd51f8a1134e67b2e40461f6b37041"
format_stamp: "Formatted at 2026-09-24 16:02:13 on dist-test-slave-n106"
server_key: "dab7c2613f43b5ffa7cca5ae79fc6108"
server_key_iv: "8c6037ec1a1cef64805815e82b601a6a"
server_key_version: "encryptionkey@0"
I20260924 16:02:13.480013 30157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:13.491581 30157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:13.494819 30157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:13.495023 30157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:13.502074 30157 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:40385
I20260924 16:02:13.502343 30224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:40385 every 8 connection(s)
I20260924 16:02:13.502777 30157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:02:13.505656 30225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:13.507889 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 30157
I20260924 16:02:13.508044 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:02:13.508311 26036 external_mini_cluster.cc:1428] Setting key f09de84b15699fd58de68f8453d64b22
I20260924 16:02:13.512569 30225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dfd51f8a1134e67b2e40461f6b37041: Bootstrap starting.
I20260924 16:02:13.515436 30225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dfd51f8a1134e67b2e40461f6b37041: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:13.516362 30225 log.cc:824] T 00000000000000000000000000000000 P 6dfd51f8a1134e67b2e40461f6b37041: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:02:13 dist-test-slave-n106 krb5kdc[30144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:02:13.518618 30225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dfd51f8a1134e67b2e40461f6b37041: No bootstrap required, opened a new log
I20260924 16:02:13.522521 30225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dfd51f8a1134e67b2e40461f6b37041 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dfd51f8a1134e67b2e40461f6b37041" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40385 } }
I20260924 16:02:13.522783 30225 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6dfd51f8a1134e67b2e40461f6b37041 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:13.522841 30225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dfd51f8a1134e67b2e40461f6b37041 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dfd51f8a1134e67b2e40461f6b37041, State: Initialized, Role: FOLLOWER
I20260924 16:02:13.523291 30225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dfd51f8a1134e67b2e40461f6b37041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dfd51f8a1134e67b2e40461f6b37041" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40385 } }
I20260924 16:02:13.523510 30225 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6dfd51f8a1134e67b2e40461f6b37041 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:02:13.523597 30225 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dfd51f8a1134e67b2e40461f6b37041 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:02:13.523728 30225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dfd51f8a1134e67b2e40461f6b37041 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:13.524024 30228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:13.509830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37048 (local address 127.25.109.62:40385)
0924 16:02:13.510279 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:13.510299 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:13.510345 (+ 46us) server_negotiation.cc:407] Connection header received
0924 16:02:13.511371 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:13.511398 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:13.511836 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:13.512222 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:13.513005 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:13.514509 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:13.515408 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:13.516117 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:13.516297 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:13.518973 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:13.518979 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:13.518993 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:13.519023 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:13.520827 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:13.521299 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:13.521305 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:13.521310 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:13.521425 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:13.521728 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:13.521731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:13.521733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:13.522153 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:13.522388 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:13.522632 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 16:02:13.522978 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":119,"threads_started":1}
I20260924 16:02:13.524843 30225 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dfd51f8a1134e67b2e40461f6b37041 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dfd51f8a1134e67b2e40461f6b37041" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40385 } }
I20260924 16:02:13.525238 30225 leader_election.cc:302] T 00000000000000000000000000000000 P 6dfd51f8a1134e67b2e40461f6b37041 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6dfd51f8a1134e67b2e40461f6b37041; no voters:
I20260924 16:02:13.525563 30225 leader_election.cc:288] T 00000000000000000000000000000000 P 6dfd51f8a1134e67b2e40461f6b37041 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:02:13.525681 30230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6dfd51f8a1134e67b2e40461f6b37041 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:13.525883 30230 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6dfd51f8a1134e67b2e40461f6b37041 [term 1 LEADER]: Becoming Leader. State: Replica: 6dfd51f8a1134e67b2e40461f6b37041, State: Running, Role: LEADER
I20260924 16:02:13.526201 30230 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dfd51f8a1134e67b2e40461f6b37041 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dfd51f8a1134e67b2e40461f6b37041" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40385 } }
I20260924 16:02:13.527007 30225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dfd51f8a1134e67b2e40461f6b37041 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:02:13.527714 30230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dfd51f8a1134e67b2e40461f6b37041 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dfd51f8a1134e67b2e40461f6b37041. Latest consensus state: current_term: 1 leader_uuid: "6dfd51f8a1134e67b2e40461f6b37041" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dfd51f8a1134e67b2e40461f6b37041" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40385 } } }
I20260924 16:02:13.527837 30230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dfd51f8a1134e67b2e40461f6b37041 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:13.528245 30237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:02:13.528343 30231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dfd51f8a1134e67b2e40461f6b37041 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6dfd51f8a1134e67b2e40461f6b37041" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dfd51f8a1134e67b2e40461f6b37041" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40385 } } }
I20260924 16:02:13.528554 30231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dfd51f8a1134e67b2e40461f6b37041 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:13.531368 30237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:02:13.537042 30237 catalog_manager.cc:1347] Generated new cluster ID: 45bf9735b1f546ae86bb9eb318789a3e
I20260924 16:02:13.537115 30237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:02:13.545392 30237 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:02:13.546638 30237 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:02:13.554567 30237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6dfd51f8a1134e67b2e40461f6b37041: Generated new TSK 0
I20260924 16:02:13.555315 30237 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
I20260924 16:02:13.587388 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:0
--local_ip_for_outbound_sockets=127.25.109.1
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:40385
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:33829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:13.701184 30251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:13.701463 30251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:13.701508 30251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:13.705766 30251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:13.705833 30251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:13.705929 30251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:02:13.711232 30251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:33829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:40385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:13.712316 30251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:13.713452 30251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:13.723104 30257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:13.723130 30256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:13.723260 30251 server_base.cc:1034] running on GCE node
W20260924 16:02:13.723104 30259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:13.723893 30251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:13.724658 30251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:13.725845 30251 hybrid_clock.cc:648] HybridClock initialized: now 1790265733725825 us; error 32 us; skew 500 ppm
Sep 24 16:02:13 dist-test-slave-n106 krb5kdc[30144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265733, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:13.728878 30251 init.cc:377] Logged in from keytab as kudu/127.25.109.1@KRBTEST.COM (short username kudu)
I20260924 16:02:13.729753 30251 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:13.729818 30251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:13.730046 30251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:13.732141 30251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-0/data/instance:
uuid: "38e9d7367a394997a9b42af976c6a747"
format_stamp: "Formatted at 2026-09-24 16:02:13 on dist-test-slave-n106"
server_key: "edbcfcf4794ff7edcf895c87b7cbf36d"
server_key_iv: "b677da2ec0856eceafe92727703328fd"
server_key_version: "encryptionkey@0"
I20260924 16:02:13.732760 30251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance:
uuid: "38e9d7367a394997a9b42af976c6a747"
format_stamp: "Formatted at 2026-09-24 16:02:13 on dist-test-slave-n106"
server_key: "edbcfcf4794ff7edcf895c87b7cbf36d"
server_key_iv: "b677da2ec0856eceafe92727703328fd"
server_key_version: "encryptionkey@0"
I20260924 16:02:13.736514 30251 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:02:13.739056 30266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:13.740324 30251 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:02:13.740494 30251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "38e9d7367a394997a9b42af976c6a747"
format_stamp: "Formatted at 2026-09-24 16:02:13 on dist-test-slave-n106"
server_key: "edbcfcf4794ff7edcf895c87b7cbf36d"
server_key_iv: "b677da2ec0856eceafe92727703328fd"
server_key_version: "encryptionkey@0"
I20260924 16:02:13.740629 30251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:13.755429 30251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:13.758662 30251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:13.758867 30251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:13.759454 30251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:13.760308 30251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:13.760380 30251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:13.760475 30251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:13.760521 30251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:13.769584 30251 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:37297
I20260924 16:02:13.769601 30380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:37297 every 8 connection(s)
I20260924 16:02:13.770203 30251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:02:13.773773 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 30251
I20260924 16:02:13.773874 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance
I20260924 16:02:13.774175 26036 external_mini_cluster.cc:1428] Setting key c796d6de5365ddc7e5a376ad9de1d947
Sep 24 16:02:13 dist-test-slave-n106 krb5kdc[30144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265733, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.109.2@KRBTEST.COM; defaulting to no policy
I20260924 16:02:13.784873 30228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:13.771947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:49965 (local address 127.25.109.62:40385)
0924 16:02:13.772088 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:13.772092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:13.773340 (+ 1248us) server_negotiation.cc:407] Connection header received
0924 16:02:13.774363 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:13.774366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:13.774429 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:13.774535 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:13.775855 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:13.776392 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:13.777646 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:13.778057 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:13.778194 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:13.780855 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:13.780862 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:13.780869 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:13.780907 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:13.782580 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:13.783170 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:13.783174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:13.783175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:13.783265 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:13.783643 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:13.783653 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:13.783655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:13.783839 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:13.783951 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:13.784550 (+ 599us) server_negotiation.cc:299] Negotiation successful
0924 16:02:13.784698 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:02:13.785696 30383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:13.772333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40385 (local address 127.25.109.1:49965)
0924 16:02:13.773196 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:13.773238 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:13.774123 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:13.774731 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:13.774740 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:13.775139 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:13.775712 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:13.775727 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:13.776573 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:13.776577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:13.777459 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:13.777469 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:13.778209 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:13.778212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:13.778358 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:13.778973 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:13.778990 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:13.780673 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:13.782731 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:13.782736 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:13.782748 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:13.783061 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:13.783362 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:13.783366 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:13.783368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:13.783520 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:13.783950 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:13.783957 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:13.784273 (+ 316us) client_negotiation.cc:769] Sending connection context
0924 16:02:13.784501 (+ 228us) client_negotiation.cc:240] Negotiation successful
0924 16:02:13.784831 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.2@KRBTEST.COM" created.
I20260924 16:02:13.786999 30381 heartbeater.cc:347] Connected to a master server at 127.25.109.62:40385
I20260924 16:02:13.787253 30381 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:13.787719 30381 heartbeater.cc:511] Master 127.25.109.62:40385 requested a full tablet report, sending...
I20260924 16:02:13.789264 30188 ts_manager.cc:194] Registered new tserver with Master: 38e9d7367a394997a9b42af976c6a747 (127.25.109.1:37297)
I20260924 16:02:13.790419 30188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.1@KRBTEST.COM'} at 127.25.109.1:49965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
I20260924 16:02:13.801235 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:0
--local_ip_for_outbound_sockets=127.25.109.2
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:40385
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:33829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:13.913214 30386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:13.913491 30386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:13.913537 30386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:13.917549 30386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:13.917609 30386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:13.917670 30386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:02:13.923017 30386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:33829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:40385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:13.924104 30386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:13.925081 30386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:13.934940 30392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:13.934913 30391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:13.934921 30394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:13.935312 30386 server_base.cc:1034] running on GCE node
I20260924 16:02:13.935750 30386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:13.936466 30386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:13.937660 30386 hybrid_clock.cc:648] HybridClock initialized: now 1790265733937647 us; error 30 us; skew 500 ppm
Sep 24 16:02:13 dist-test-slave-n106 krb5kdc[30144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265733, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:13.940466 30386 init.cc:377] Logged in from keytab as kudu/127.25.109.2@KRBTEST.COM (short username kudu)
I20260924 16:02:13.941206 30386 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:13.941303 30386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:13.941540 30386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:13.943675 30386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-1/data/instance:
uuid: "bc49969da30a40079f8ebe29d9dd4050"
format_stamp: "Formatted at 2026-09-24 16:02:13 on dist-test-slave-n106"
server_key: "964c45bfa9dd74e824f317fc54efd247"
server_key_iv: "bbf58b46a905cba673fa7796f08289b9"
server_key_version: "encryptionkey@0"
I20260924 16:02:13.944258 30386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance:
uuid: "bc49969da30a40079f8ebe29d9dd4050"
format_stamp: "Formatted at 2026-09-24 16:02:13 on dist-test-slave-n106"
server_key: "964c45bfa9dd74e824f317fc54efd247"
server_key_iv: "bbf58b46a905cba673fa7796f08289b9"
server_key_version: "encryptionkey@0"
I20260924 16:02:13.947955 30386 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:02:13.950454 30400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:13.951645 30386 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:02:13.951790 30386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "bc49969da30a40079f8ebe29d9dd4050"
format_stamp: "Formatted at 2026-09-24 16:02:13 on dist-test-slave-n106"
server_key: "964c45bfa9dd74e824f317fc54efd247"
server_key_iv: "bbf58b46a905cba673fa7796f08289b9"
server_key_version: "encryptionkey@0"
I20260924 16:02:13.951900 30386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:13.971095 30386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:13.974385 30386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:13.974591 30386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:13.975183 30386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:13.976011 30386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:13.976082 30386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:13.976184 30386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:13.976235 30386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:13.985355 30386 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:44853
I20260924 16:02:13.985369 30515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:44853 every 8 connection(s)
I20260924 16:02:13.985934 30386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:02:13.987502 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 30386
I20260924 16:02:13.987637 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance
I20260924 16:02:13.987954 26036 external_mini_cluster.cc:1428] Setting key bc666f9583f75ec20ed93dd67ec5f86d
Sep 24 16:02:13 dist-test-slave-n106 krb5kdc[30144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265733, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.3@KRBTEST.COM" created.
I20260924 16:02:14.000249 30228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:13.987763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:56445 (local address 127.25.109.62:40385)
0924 16:02:13.987899 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:13.987904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:13.989474 (+ 1570us) server_negotiation.cc:407] Connection header received
0924 16:02:13.990350 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:13.990354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:13.990408 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:13.990497 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:13.991643 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:13.992247 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:13.993565 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:13.993955 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:13.994088 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:13.996475 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:13.996479 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:13.996484 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:13.996518 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:13.998086 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:13.998666 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:13.998669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:13.998671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:13.998748 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:13.999050 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:13.999053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:13.999054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:13.999239 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:13.999315 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:13.999972 (+ 657us) server_negotiation.cc:299] Negotiation successful
0924 16:02:14.000127 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 16:02:14.001382 30518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:13.988239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40385 (local address 127.25.109.2:56445)
0924 16:02:13.989312 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:13.989368 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:13.990148 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:13.990681 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:13.990688 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:13.991028 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:13.991489 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:13.991503 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:13.992405 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:13.992410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:13.993414 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:13.993430 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:13.994114 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:13.994116 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:13.994255 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:13.994853 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:13.994869 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:13.996264 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:13.998196 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:13.998201 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:13.998212 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:13.998577 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:13.998832 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:13.998834 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:13.998836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:13.998967 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:13.999348 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:13.999353 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:13.999650 (+ 297us) client_negotiation.cc:769] Sending connection context
0924 16:02:13.999905 (+ 255us) client_negotiation.cc:240] Negotiation successful
0924 16:02:14.000252 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":240,"threads_started":1}
I20260924 16:02:14.002619 30516 heartbeater.cc:347] Connected to a master server at 127.25.109.62:40385
I20260924 16:02:14.002869 30516 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:14.003372 30516 heartbeater.cc:511] Master 127.25.109.62:40385 requested a full tablet report, sending...
I20260924 16:02:14.004632 30188 ts_manager.cc:194] Registered new tserver with Master: bc49969da30a40079f8ebe29d9dd4050 (127.25.109.2:44853)
I20260924 16:02:14.005225 30188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.2@KRBTEST.COM'} at 127.25.109.2:56445
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
I20260924 16:02:14.015372 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:0
--local_ip_for_outbound_sockets=127.25.109.3
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:40385
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:33829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:14.126943 30521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:14.127209 30521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:14.127277 30521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:14.131763 30521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:14.131851 30521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:14.131942 30521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:02:14.137308 30521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:33829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:40385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:14.138656 30521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:14.139736 30521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:02:14.149730 30521 server_base.cc:1034] running on GCE node
W20260924 16:02:14.149658 30529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:14.149658 30526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:14.149653 30527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:14.150365 30521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:14.151042 30521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:14.152243 30521 hybrid_clock.cc:648] HybridClock initialized: now 1790265734152228 us; error 25 us; skew 500 ppm
Sep 24 16:02:14 dist-test-slave-n106 krb5kdc[30144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265734, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:14.155056 30521 init.cc:377] Logged in from keytab as kudu/127.25.109.3@KRBTEST.COM (short username kudu)
I20260924 16:02:14.155874 30521 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:14.155966 30521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:14.156203 30521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:14.158336 30521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-2/data/instance:
uuid: "585873e8ea574e5cba806115620e7d54"
format_stamp: "Formatted at 2026-09-24 16:02:14 on dist-test-slave-n106"
server_key: "e49fb92e82cd075aafae6ee0a4980892"
server_key_iv: "9fd3cf219870ee1eecfa2073209603f4"
server_key_version: "encryptionkey@0"
I20260924 16:02:14.158922 30521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance:
uuid: "585873e8ea574e5cba806115620e7d54"
format_stamp: "Formatted at 2026-09-24 16:02:14 on dist-test-slave-n106"
server_key: "e49fb92e82cd075aafae6ee0a4980892"
server_key_iv: "9fd3cf219870ee1eecfa2073209603f4"
server_key_version: "encryptionkey@0"
I20260924 16:02:14.162632 30521 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 16:02:14.165097 30535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:14.166406 30521 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:02:14.166576 30521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "585873e8ea574e5cba806115620e7d54"
format_stamp: "Formatted at 2026-09-24 16:02:14 on dist-test-slave-n106"
server_key: "e49fb92e82cd075aafae6ee0a4980892"
server_key_iv: "9fd3cf219870ee1eecfa2073209603f4"
server_key_version: "encryptionkey@0"
I20260924 16:02:14.166693 30521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:14.182019 30521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:14.185247 30521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:14.185449 30521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:14.186024 30521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:14.186848 30521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:14.186929 30521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:14.187004 30521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:14.187042 30521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:14.195475 30521 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:39833
I20260924 16:02:14.195490 30649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:39833 every 8 connection(s)
I20260924 16:02:14.196101 30521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:02:14.201472 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 30521
I20260924 16:02:14.201627 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance
I20260924 16:02:14.201913 26036 external_mini_cluster.cc:1428] Setting key ceb59304a8e72d70858444ca8eb222b8
Sep 24 16:02:14 dist-test-slave-n106 krb5kdc[30144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265734, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:02:14.209473 30228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:14.197968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:38179 (local address 127.25.109.62:40385)
0924 16:02:14.198130 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:14.198134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:14.199272 (+ 1138us) server_negotiation.cc:407] Connection header received
0924 16:02:14.200177 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:14.200179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:14.200241 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:14.200352 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:14.201547 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:14.202214 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:14.203342 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:14.203582 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:14.203699 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:14.206030 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:14.206034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:14.206037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:14.206070 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:14.207445 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:14.207957 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:14.207961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:14.207963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:14.208048 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:14.208397 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:14.208401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:14.208419 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:14.208603 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:14.208715 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:14.209192 (+ 477us) server_negotiation.cc:299] Negotiation successful
0924 16:02:14.209309 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 16:02:14.210206 30652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:14.198330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40385 (local address 127.25.109.3:38179)
0924 16:02:14.199120 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:14.199159 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:14.199958 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:14.200586 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:14.200594 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:14.200935 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:14.201393 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:14.201406 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:14.202340 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:14.202344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:14.203211 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:14.203221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:14.203733 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:14.203737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:14.203911 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:14.204467 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:14.204485 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:14.205870 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:14.207566 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:14.207572 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:14.207586 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:14.207848 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:14.208153 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:14.208156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:14.208158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:14.208295 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:14.208713 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:14.208720 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:14.208954 (+ 234us) client_negotiation.cc:769] Sending connection context
0924 16:02:14.209145 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 16:02:14.209443 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":126,"threads_started":1}
I20260924 16:02:14.211261 30650 heartbeater.cc:347] Connected to a master server at 127.25.109.62:40385
I20260924 16:02:14.211462 30650 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:14.211848 30650 heartbeater.cc:511] Master 127.25.109.62:40385 requested a full tablet report, sending...
I20260924 16:02:14.212944 30188 ts_manager.cc:194] Registered new tserver with Master: 585873e8ea574e5cba806115620e7d54 (127.25.109.3:39833)
I20260924 16:02:14.213508 30188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.3@KRBTEST.COM'} at 127.25.109.3:38179
I20260924 16:02:14.215387 26036 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:02:14.223942 30228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:14.217207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37062 (local address 127.25.109.62:40385)
0924 16:02:14.217346 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:14.217351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:14.217554 (+ 203us) server_negotiation.cc:407] Connection header received
0924 16:02:14.217643 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:14.217645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:14.217681 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:14.217744 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:14.218313 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:14.218783 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:14.219613 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:14.219808 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:14.219879 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:14.221193 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:14.221196 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:14.221199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:14.221218 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:14.222386 (+ 1168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:14.222756 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:14.222759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:14.222760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:14.222830 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:14.223110 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:14.223112 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:14.223113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:14.223300 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:14.223401 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:14.223650 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 16:02:14.223737 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 16:02:14.226858 30188 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:02:14.230127 30188 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:02:14.244257 30662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:14.238580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42640 (local address 127.25.109.3:39833)
0924 16:02:14.238889 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:14.238906 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:14.238945 (+ 39us) server_negotiation.cc:407] Connection header received
0924 16:02:14.239055 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:14.239061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:14.239281 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:14.239441 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:14.239740 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:14.240928 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:14.242615 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:14.243566 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:14.243684 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:14.243829 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:14.244013 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 16:02:14.244103 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":122,"threads_started":1}
I20260924 16:02:14.244251 30661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:14.238428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:39833 (local address 127.0.0.1:42640)
0924 16:02:14.238749 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:14.238795 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:14.238913 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:14.239449 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:14.239454 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:14.239477 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:14.239605 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:14.239612 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:14.241061 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:14.241065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:14.242459 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:14.242469 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:14.243687 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:14.243690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:14.243795 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:14.243831 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 16:02:14.243973 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 16:02:14.244094 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":18,"thread_start_us":89,"threads_started":1}
I20260924 16:02:14.244822 30663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:14.239516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:37297 (local address 127.0.0.1:44472)
0924 16:02:14.239805 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:14.239820 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:14.239913 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:14.240786 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:14.240791 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:14.240808 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:14.240905 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:14.240911 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:14.242337 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:14.242340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:14.243341 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:14.243353 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:14.244391 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:14.244397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:14.244522 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:14.244547 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 16:02:14.244596 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 16:02:14.244677 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":20,"thread_start_us":91,"threads_started":1}
I20260924 16:02:14.244983 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:14.239195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:44853 (local address 127.0.0.1:51810)
0924 16:02:14.239740 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:14.239754 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:14.239831 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:14.240731 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:14.240734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:14.240753 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:14.241027 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:14.241033 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:14.242001 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:14.242006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:14.243173 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:14.243185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:14.244547 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:14.244550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:14.244649 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:14.244667 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 16:02:14.244723 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 16:02:14.244777 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":45,"thread_start_us":94,"threads_started":1}
I20260924 16:02:14.245501 30664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:14.239353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51810 (local address 127.25.109.2:44853)
0924 16:02:14.240262 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:14.240273 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:14.240295 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:02:14.240370 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:14.240375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:14.240584 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:14.240739 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:14.241139 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:14.241868 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:14.243310 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:14.244151 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:14.244260 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:14.245124 (+ 864us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:14.245179 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 16:02:14.245281 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":787,"thread_start_us":122,"threads_started":1}
I20260924 16:02:14.245757 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:14.240053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44472 (local address 127.25.109.1:37297)
0924 16:02:14.240372 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:14.240378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:14.240395 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:02:14.240481 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:14.240486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:14.240638 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:14.240906 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:14.241069 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:14.242184 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:14.243511 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:14.244256 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:14.245338 (+ 1082us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:14.245483 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:14.245533 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 16:02:14.245615 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":129,"threads_started":1}
I20260924 16:02:14.247484 30584 tablet_service.cc:1467] Processing CreateTablet for tablet b8ca05323da846e4912a4a4d4903d7b3 (DEFAULT_TABLE table=test-table [id=a4ee10c2d36f44f98eeedf83bf441ecf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:14.248555 30584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8ca05323da846e4912a4a4d4903d7b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:14.248296 30315 tablet_service.cc:1467] Processing CreateTablet for tablet b8ca05323da846e4912a4a4d4903d7b3 (DEFAULT_TABLE table=test-table [id=a4ee10c2d36f44f98eeedf83bf441ecf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:14.248292 30450 tablet_service.cc:1467] Processing CreateTablet for tablet b8ca05323da846e4912a4a4d4903d7b3 (DEFAULT_TABLE table=test-table [id=a4ee10c2d36f44f98eeedf83bf441ecf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:14.249343 30450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8ca05323da846e4912a4a4d4903d7b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:14.249343 30315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8ca05323da846e4912a4a4d4903d7b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:14.254640 30667 tablet_bootstrap.cc:492] T b8ca05323da846e4912a4a4d4903d7b3 P 585873e8ea574e5cba806115620e7d54: Bootstrap starting.
I20260924 16:02:14.256502 30668 tablet_bootstrap.cc:492] T b8ca05323da846e4912a4a4d4903d7b3 P 38e9d7367a394997a9b42af976c6a747: Bootstrap starting.
I20260924 16:02:14.257292 30669 tablet_bootstrap.cc:492] T b8ca05323da846e4912a4a4d4903d7b3 P bc49969da30a40079f8ebe29d9dd4050: Bootstrap starting.
I20260924 16:02:14.257494 30667 tablet_bootstrap.cc:654] T b8ca05323da846e4912a4a4d4903d7b3 P 585873e8ea574e5cba806115620e7d54: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:14.258752 30668 tablet_bootstrap.cc:654] T b8ca05323da846e4912a4a4d4903d7b3 P 38e9d7367a394997a9b42af976c6a747: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:14.258760 30667 log.cc:824] T b8ca05323da846e4912a4a4d4903d7b3 P 585873e8ea574e5cba806115620e7d54: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:14.259011 30669 tablet_bootstrap.cc:654] T b8ca05323da846e4912a4a4d4903d7b3 P bc49969da30a40079f8ebe29d9dd4050: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:14.259845 30668 log.cc:824] T b8ca05323da846e4912a4a4d4903d7b3 P 38e9d7367a394997a9b42af976c6a747: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:14.260133 30669 log.cc:824] T b8ca05323da846e4912a4a4d4903d7b3 P bc49969da30a40079f8ebe29d9dd4050: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:14.261166 30667 tablet_bootstrap.cc:492] T b8ca05323da846e4912a4a4d4903d7b3 P 585873e8ea574e5cba806115620e7d54: No bootstrap required, opened a new log
I20260924 16:02:14.261382 30667 ts_tablet_manager.cc:1397] T b8ca05323da846e4912a4a4d4903d7b3 P 585873e8ea574e5cba806115620e7d54: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 16:02:14.261864 30668 tablet_bootstrap.cc:492] T b8ca05323da846e4912a4a4d4903d7b3 P 38e9d7367a394997a9b42af976c6a747: No bootstrap required, opened a new log
I20260924 16:02:14.261864 30669 tablet_bootstrap.cc:492] T b8ca05323da846e4912a4a4d4903d7b3 P bc49969da30a40079f8ebe29d9dd4050: No bootstrap required, opened a new log
I20260924 16:02:14.262053 30668 ts_tablet_manager.cc:1397] T b8ca05323da846e4912a4a4d4903d7b3 P 38e9d7367a394997a9b42af976c6a747: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 16:02:14.262053 30669 ts_tablet_manager.cc:1397] T b8ca05323da846e4912a4a4d4903d7b3 P bc49969da30a40079f8ebe29d9dd4050: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260924 16:02:14.264674 30667 raft_consensus.cc:348] T b8ca05323da846e4912a4a4d4903d7b3 P 585873e8ea574e5cba806115620e7d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585873e8ea574e5cba806115620e7d54" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 39833 } } peers { permanent_uuid: "bc49969da30a40079f8ebe29d9dd4050" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 44853 } } peers { permanent_uuid: "38e9d7367a394997a9b42af976c6a747" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37297 } }
I20260924 16:02:14.264918 30667 raft_consensus.cc:374] T b8ca05323da846e4912a4a4d4903d7b3 P 585873e8ea574e5cba806115620e7d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:14.264976 30667 raft_consensus.cc:729] T b8ca05323da846e4912a4a4d4903d7b3 P 585873e8ea574e5cba806115620e7d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 585873e8ea574e5cba806115620e7d54, State: Initialized, Role: FOLLOWER
I20260924 16:02:14.265403 30667 consensus_queue.cc:260] T b8ca05323da846e4912a4a4d4903d7b3 P 585873e8ea574e5cba806115620e7d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585873e8ea574e5cba806115620e7d54" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 39833 } } peers { permanent_uuid: "bc49969da30a40079f8ebe29d9dd4050" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 44853 } } peers { permanent_uuid: "38e9d7367a394997a9b42af976c6a747" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37297 } }
I20260924 16:02:14.266050 30650 heartbeater.cc:503] Master 127.25.109.62:40385 was elected leader, sending a full tablet report...
I20260924 16:02:14.266058 30669 raft_consensus.cc:348] T b8ca05323da846e4912a4a4d4903d7b3 P bc49969da30a40079f8ebe29d9dd4050 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585873e8ea574e5cba806115620e7d54" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 39833 } } peers { permanent_uuid: "bc49969da30a40079f8ebe29d9dd4050" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 44853 } } peers { permanent_uuid: "38e9d7367a394997a9b42af976c6a747" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37297 } }
I20260924 16:02:14.266058 30668 raft_consensus.cc:348] T b8ca05323da846e4912a4a4d4903d7b3 P 38e9d7367a394997a9b42af976c6a747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585873e8ea574e5cba806115620e7d54" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 39833 } } peers { permanent_uuid: "bc49969da30a40079f8ebe29d9dd4050" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 44853 } } peers { permanent_uuid: "38e9d7367a394997a9b42af976c6a747" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37297 } }
I20260924 16:02:14.266307 30668 raft_consensus.cc:374] T b8ca05323da846e4912a4a4d4903d7b3 P 38e9d7367a394997a9b42af976c6a747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:14.266306 30669 raft_consensus.cc:374] T b8ca05323da846e4912a4a4d4903d7b3 P bc49969da30a40079f8ebe29d9dd4050 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:14.266366 30669 raft_consensus.cc:729] T b8ca05323da846e4912a4a4d4903d7b3 P bc49969da30a40079f8ebe29d9dd4050 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc49969da30a40079f8ebe29d9dd4050, State: Initialized, Role: FOLLOWER
I20260924 16:02:14.266366 30668 raft_consensus.cc:729] T b8ca05323da846e4912a4a4d4903d7b3 P 38e9d7367a394997a9b42af976c6a747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38e9d7367a394997a9b42af976c6a747, State: Initialized, Role: FOLLOWER
I20260924 16:02:14.266417 30667 ts_tablet_manager.cc:1428] T b8ca05323da846e4912a4a4d4903d7b3 P 585873e8ea574e5cba806115620e7d54: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 16:02:14.266815 30668 consensus_queue.cc:260] T b8ca05323da846e4912a4a4d4903d7b3 P 38e9d7367a394997a9b42af976c6a747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585873e8ea574e5cba806115620e7d54" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 39833 } } peers { permanent_uuid: "bc49969da30a40079f8ebe29d9dd4050" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 44853 } } peers { permanent_uuid: "38e9d7367a394997a9b42af976c6a747" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37297 } }
I20260924 16:02:14.266815 30669 consensus_queue.cc:260] T b8ca05323da846e4912a4a4d4903d7b3 P bc49969da30a40079f8ebe29d9dd4050 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585873e8ea574e5cba806115620e7d54" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 39833 } } peers { permanent_uuid: "bc49969da30a40079f8ebe29d9dd4050" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 44853 } } peers { permanent_uuid: "38e9d7367a394997a9b42af976c6a747" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37297 } }
I20260924 16:02:14.267653 30516 heartbeater.cc:503] Master 127.25.109.62:40385 was elected leader, sending a full tablet report...
I20260924 16:02:14.267768 30669 ts_tablet_manager.cc:1428] T b8ca05323da846e4912a4a4d4903d7b3 P bc49969da30a40079f8ebe29d9dd4050: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260924 16:02:14.267771 30668 ts_tablet_manager.cc:1428] T b8ca05323da846e4912a4a4d4903d7b3 P 38e9d7367a394997a9b42af976c6a747: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 16:02:14.268610 30381 heartbeater.cc:503] Master 127.25.109.62:40385 was elected leader, sending a full tablet report...
W20260924 16:02:14.271492 30382 tablet.cc:2369] T b8ca05323da846e4912a4a4d4903d7b3 P 38e9d7367a394997a9b42af976c6a747: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:02:14.447464 30651 tablet.cc:2369] T b8ca05323da846e4912a4a4d4903d7b3 P 585873e8ea574e5cba806115620e7d54: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:02:14.449831 30675 raft_consensus.cc:482] T b8ca05323da846e4912a4a4d4903d7b3 P bc49969da30a40079f8ebe29d9dd4050 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:02:14.450057 30675 raft_consensus.cc:504] T b8ca05323da846e4912a4a4d4903d7b3 P bc49969da30a40079f8ebe29d9dd4050 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585873e8ea574e5cba806115620e7d54" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 39833 } } peers { permanent_uuid: "bc49969da30a40079f8ebe29d9dd4050" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 44853 } } peers { permanent_uuid: "38e9d7367a394997a9b42af976c6a747" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37297 } }
I20260924 16:02:14.451682 30675 leader_election.cc:288] T b8ca05323da846e4912a4a4d4903d7b3 P bc49969da30a40079f8ebe29d9dd4050 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 585873e8ea574e5cba806115620e7d54 (127.25.109.3:39833), 38e9d7367a394997a9b42af976c6a747 (127.25.109.1:37297)
I20260924 16:02:14.455538 30662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:14.452026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:49449 (local address 127.25.109.3:39833)
0924 16:02:14.452152 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:14.452159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:14.452199 (+ 40us) server_negotiation.cc:407] Connection header received
0924 16:02:14.452272 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:14.452278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:14.452328 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:14.452658 (+ 330us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:14.452970 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:14.453485 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:14.454542 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:14.454968 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:14.455098 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:14.455153 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:14.455345 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 16:02:14.455414 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 16:02:14.455538 30518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:14.451917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:39833 (local address 127.25.109.2:49449)
0924 16:02:14.452052 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:14.452070 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:14.452182 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:14.452690 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:14.452694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:14.452717 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:14.452846 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:14.452853 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:14.453597 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:14.453602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:14.454409 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:14.454418 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:14.455090 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:14.455094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:14.455208 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:14.455237 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 16:02:14.455327 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 16:02:14.455402 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 16:02:14.456058 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:14.452374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:60149 (local address 127.25.109.1:37297)
0924 16:02:14.452546 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:14.452552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:14.452780 (+ 228us) server_negotiation.cc:407] Connection header received
0924 16:02:14.452959 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:14.452962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:14.453005 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:14.453099 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:14.453358 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:14.453847 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:14.454951 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:14.455343 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:14.455497 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:14.455547 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:14.455853 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 16:02:14.455926 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 16:02:14.456082 30676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:14.452239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:37297 (local address 127.25.109.2:60149)
0924 16:02:14.452706 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:14.452719 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:14.452848 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:14.453109 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:14.453113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:14.453127 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:14.453254 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:14.453259 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:14.453956 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:14.453960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:14.454833 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:14.454844 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:14.455629 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:14.455633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:14.455734 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:14.455756 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 16:02:14.455826 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 16:02:14.455933 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":91,"threads_started":1}
I20260924 16:02:14.456321 30604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ca05323da846e4912a4a4d4903d7b3" candidate_uuid: "bc49969da30a40079f8ebe29d9dd4050" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "585873e8ea574e5cba806115620e7d54" is_pre_election: true
I20260924 16:02:14.456681 30604 raft_consensus.cc:2454] T b8ca05323da846e4912a4a4d4903d7b3 P 585873e8ea574e5cba806115620e7d54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc49969da30a40079f8ebe29d9dd4050 in term 0.
I20260924 16:02:14.457016 30335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ca05323da846e4912a4a4d4903d7b3" candidate_uuid: "bc49969da30a40079f8ebe29d9dd4050" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38e9d7367a394997a9b42af976c6a747" is_pre_election: true
I20260924 16:02:14.457247 30401 leader_election.cc:302] T b8ca05323da846e4912a4a4d4903d7b3 P bc49969da30a40079f8ebe29d9dd4050 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585873e8ea574e5cba806115620e7d54, bc49969da30a40079f8ebe29d9dd4050; no voters:
I20260924 16:02:14.457336 30335 raft_consensus.cc:2454] T b8ca05323da846e4912a4a4d4903d7b3 P 38e9d7367a394997a9b42af976c6a747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc49969da30a40079f8ebe29d9dd4050 in term 0.
I20260924 16:02:14.457516 30675 raft_consensus.cc:2781] T b8ca05323da846e4912a4a4d4903d7b3 P bc49969da30a40079f8ebe29d9dd4050 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:02:14.457603 30675 raft_consensus.cc:482] T b8ca05323da846e4912a4a4d4903d7b3 P bc49969da30a40079f8ebe29d9dd4050 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:02:14.457670 30675 raft_consensus.cc:3037] T b8ca05323da846e4912a4a4d4903d7b3 P bc49969da30a40079f8ebe29d9dd4050 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:14.458760 30675 raft_consensus.cc:504] T b8ca05323da846e4912a4a4d4903d7b3 P bc49969da30a40079f8ebe29d9dd4050 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585873e8ea574e5cba806115620e7d54" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 39833 } } peers { permanent_uuid: "bc49969da30a40079f8ebe29d9dd4050" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 44853 } } peers { permanent_uuid: "38e9d7367a394997a9b42af976c6a747" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37297 } }
I20260924 16:02:14.459113 30675 leader_election.cc:288] T b8ca05323da846e4912a4a4d4903d7b3 P bc49969da30a40079f8ebe29d9dd4050 [CANDIDATE]: Term 1 election: Requested vote from peers 585873e8ea574e5cba806115620e7d54 (127.25.109.3:39833), 38e9d7367a394997a9b42af976c6a747 (127.25.109.1:37297)
I20260924 16:02:14.459501 30604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ca05323da846e4912a4a4d4903d7b3" candidate_uuid: "bc49969da30a40079f8ebe29d9dd4050" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "585873e8ea574e5cba806115620e7d54"
I20260924 16:02:14.459510 30335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ca05323da846e4912a4a4d4903d7b3" candidate_uuid: "bc49969da30a40079f8ebe29d9dd4050" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38e9d7367a394997a9b42af976c6a747"
I20260924 16:02:14.459628 30604 raft_consensus.cc:3037] T b8ca05323da846e4912a4a4d4903d7b3 P 585873e8ea574e5cba806115620e7d54 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:14.459641 30335 raft_consensus.cc:3037] T b8ca05323da846e4912a4a4d4903d7b3 P 38e9d7367a394997a9b42af976c6a747 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:14.460938 30335 raft_consensus.cc:2454] T b8ca05323da846e4912a4a4d4903d7b3 P 38e9d7367a394997a9b42af976c6a747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc49969da30a40079f8ebe29d9dd4050 in term 1.
I20260924 16:02:14.460938 30604 raft_consensus.cc:2454] T b8ca05323da846e4912a4a4d4903d7b3 P 585873e8ea574e5cba806115620e7d54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc49969da30a40079f8ebe29d9dd4050 in term 1.
I20260924 16:02:14.461251 30401 leader_election.cc:302] T b8ca05323da846e4912a4a4d4903d7b3 P bc49969da30a40079f8ebe29d9dd4050 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38e9d7367a394997a9b42af976c6a747, bc49969da30a40079f8ebe29d9dd4050; no voters:
I20260924 16:02:14.461417 30675 raft_consensus.cc:2781] T b8ca05323da846e4912a4a4d4903d7b3 P bc49969da30a40079f8ebe29d9dd4050 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:14.461629 30675 raft_consensus.cc:686] T b8ca05323da846e4912a4a4d4903d7b3 P bc49969da30a40079f8ebe29d9dd4050 [term 1 LEADER]: Becoming Leader. State: Replica: bc49969da30a40079f8ebe29d9dd4050, State: Running, Role: LEADER
I20260924 16:02:14.461967 30675 consensus_queue.cc:237] T b8ca05323da846e4912a4a4d4903d7b3 P bc49969da30a40079f8ebe29d9dd4050 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585873e8ea574e5cba806115620e7d54" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 39833 } } peers { permanent_uuid: "bc49969da30a40079f8ebe29d9dd4050" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 44853 } } peers { permanent_uuid: "38e9d7367a394997a9b42af976c6a747" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37297 } }
I20260924 16:02:14.465360 30187 catalog_manager.cc:5495] T b8ca05323da846e4912a4a4d4903d7b3 P bc49969da30a40079f8ebe29d9dd4050 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc49969da30a40079f8ebe29d9dd4050 (127.25.109.2). New cstate: current_term: 1 leader_uuid: "bc49969da30a40079f8ebe29d9dd4050" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585873e8ea574e5cba806115620e7d54" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 39833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc49969da30a40079f8ebe29d9dd4050" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 44853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38e9d7367a394997a9b42af976c6a747" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37297 } health_report { overall_health: UNKNOWN } } }
I20260924 16:02:14.482350 30664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:14.478795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60658 (local address 127.25.109.2:44853)
0924 16:02:14.478919 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:14.478923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:14.478935 (+ 12us) server_negotiation.cc:407] Connection header received
0924 16:02:14.479156 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:14.479158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:14.479216 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:14.479328 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:02:14.479624 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:14.480119 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:14.481050 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:14.481264 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:14.481350 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:14.481633 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:02:14.482043 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:02:14.482144 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:14.482193 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 16:02:14.482241 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 16:02:14.486961 30517 tablet.cc:2369] T b8ca05323da846e4912a4a4d4903d7b3 P bc49969da30a40079f8ebe29d9dd4050: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:02:14.488647 30604 raft_consensus.cc:1261] T b8ca05323da846e4912a4a4d4903d7b3 P 585873e8ea574e5cba806115620e7d54 [term 1 FOLLOWER]: Refusing update from remote peer bc49969da30a40079f8ebe29d9dd4050: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:02:14.488703 30335 raft_consensus.cc:1261] T b8ca05323da846e4912a4a4d4903d7b3 P 38e9d7367a394997a9b42af976c6a747 [term 1 FOLLOWER]: Refusing update from remote peer bc49969da30a40079f8ebe29d9dd4050: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:02:14.489382 30677 consensus_queue.cc:1035] T b8ca05323da846e4912a4a4d4903d7b3 P bc49969da30a40079f8ebe29d9dd4050 [LEADER]: Connected to new peer: Peer: permanent_uuid: "585873e8ea574e5cba806115620e7d54" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 39833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:14.489571 30675 consensus_queue.cc:1035] T b8ca05323da846e4912a4a4d4903d7b3 P bc49969da30a40079f8ebe29d9dd4050 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38e9d7367a394997a9b42af976c6a747" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:14.496383 30680 mvcc.cc:205] Tried to move back new op lower bound from 7332928448457322496 to 7332928448358854656. Current Snapshot: MvccSnapshot[applied={T|T < 7332928448457322496}]
I20260924 16:02:14.498485 30681 mvcc.cc:205] Tried to move back new op lower bound from 7332928448457322496 to 7332928448358854656. Current Snapshot: MvccSnapshot[applied={T|T < 7332928448457322496}]
I20260924 16:02:14.505523 30187 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:02:14.505689 30187 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:02:14.507993 30187 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6dfd51f8a1134e67b2e40461f6b37041: Sending DeleteTablet for 3 replicas of tablet b8ca05323da846e4912a4a4d4903d7b3
I20260924 16:02:14.508787 30584 tablet_service.cc:1514] Processing DeleteTablet for tablet b8ca05323da846e4912a4a4d4903d7b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:14 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:42640
I20260924 16:02:14.509326 30450 tablet_service.cc:1514] Processing DeleteTablet for tablet b8ca05323da846e4912a4a4d4903d7b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:14 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:51810
I20260924 16:02:14.509493 30315 tablet_service.cc:1514] Processing DeleteTablet for tablet b8ca05323da846e4912a4a4d4903d7b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:14 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:44472
I20260924 16:02:14.509943 30692 tablet_replica.cc:330] T b8ca05323da846e4912a4a4d4903d7b3 P 38e9d7367a394997a9b42af976c6a747: stopping tablet replica
I20260924 16:02:14.510289 30692 raft_consensus.cc:2229] T b8ca05323da846e4912a4a4d4903d7b3 P 38e9d7367a394997a9b42af976c6a747 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:14.510613 30692 raft_consensus.cc:2258] T b8ca05323da846e4912a4a4d4903d7b3 P 38e9d7367a394997a9b42af976c6a747 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:14.510769 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 30251
I20260924 16:02:14.516700 30691 tablet_replica.cc:330] T b8ca05323da846e4912a4a4d4903d7b3 P bc49969da30a40079f8ebe29d9dd4050: stopping tablet replica
I20260924 16:02:14.517079 30691 raft_consensus.cc:2229] T b8ca05323da846e4912a4a4d4903d7b3 P bc49969da30a40079f8ebe29d9dd4050 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:02:14.517575 30691 raft_consensus.cc:2258] T b8ca05323da846e4912a4a4d4903d7b3 P bc49969da30a40079f8ebe29d9dd4050 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:14.518038 30690 tablet_replica.cc:330] T b8ca05323da846e4912a4a4d4903d7b3 P 585873e8ea574e5cba806115620e7d54: stopping tablet replica
I20260924 16:02:14.518370 30690 raft_consensus.cc:2229] T b8ca05323da846e4912a4a4d4903d7b3 P 585873e8ea574e5cba806115620e7d54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:14.518719 30690 raft_consensus.cc:2258] T b8ca05323da846e4912a4a4d4903d7b3 P 585873e8ea574e5cba806115620e7d54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:14.519229 30691 ts_tablet_manager.cc:1905] T b8ca05323da846e4912a4a4d4903d7b3 P bc49969da30a40079f8ebe29d9dd4050: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 16:02:14.519477 30172 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.109.1:37297 (error 108)
I20260924 16:02:14.520056 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 30386
I20260924 16:02:14.520905 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:14.520225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:37297 (local address 127.0.0.1:34254)
0924 16:02:14.520717 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:14.520801 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.1:37297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385}
W20260924 16:02:14.521509 30172 catalog_manager.cc:4584] TS 38e9d7367a394997a9b42af976c6a747 (127.25.109.1:37297): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b8ca05323da846e4912a4a4d4903d7b3: Network error: Client connection negotiation failed: client connection to 127.25.109.1:37297: connect: Connection refused (error 111)
I20260924 16:02:14.522624 30690 ts_tablet_manager.cc:1905] T b8ca05323da846e4912a4a4d4903d7b3 P 585873e8ea574e5cba806115620e7d54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:14.524850 30690 ts_tablet_manager.cc:1918] T b8ca05323da846e4912a4a4d4903d7b3 P 585873e8ea574e5cba806115620e7d54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:02:14.524963 30690 log.cc:1196] T b8ca05323da846e4912a4a4d4903d7b3 P 585873e8ea574e5cba806115620e7d54: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265708917108-26036-0/minicluster-data/ts-2/wal/wals/b8ca05323da846e4912a4a4d4903d7b3
I20260924 16:02:14.525264 30690 ts_tablet_manager.cc:1939] T b8ca05323da846e4912a4a4d4903d7b3 P 585873e8ea574e5cba806115620e7d54: Deleting consensus metadata
I20260924 16:02:14.526316 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 30521
I20260924 16:02:14.527447 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:14.527107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:44853 (local address 127.0.0.1:60670)
0924 16:02:14.527292 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:14.527355 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.2:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260924 16:02:14.532264 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 30157
2026-09-24T16:02:14Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2368 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T16:02:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:02:14Z Disabled control of system clock
I20260924 16:02:14.562548 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:44639
--webserver_interface=127.25.109.62
--webserver_port=0
--builtin_ntp_servers=127.25.109.20:43079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:44639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 16:02:14.673942 30697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:14.674227 30697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:14.674265 30697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:14.678483 30697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:02:14.678547 30697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:14.678563 30697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:14.678581 30697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:02:14.678596 30697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:02:14.684013 30697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:43079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:44639
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:44639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:14.685410 30697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:14.686863 30697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:14.695663 30705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:14.695657 30702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:14.695775 30697 server_base.cc:1034] running on GCE node
W20260924 16:02:14.695657 30703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:14.696691 30697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:14.697862 30697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:14.699054 30697 hybrid_clock.cc:648] HybridClock initialized: now 1790265734699029 us; error 50 us; skew 500 ppm
I20260924 16:02:14.701023 30697 webserver.cc:460] Webserver started at http://127.25.109.62:41865/ using document root <none> and password file <none>
I20260924 16:02:14.701632 30697 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:14.701728 30697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:14.701972 30697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:14.704061 30697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "f919079462d04b38a282e3e919d43d24"
format_stamp: "Formatted at 2026-09-24 16:02:14 on dist-test-slave-n106"
server_key: "074011dbb7fb090ad708e570ea22287a"
server_key_iv: "b71cc5e630c14ce2e6998992af2bc549"
server_key_version: "encryptionkey@0"
I20260924 16:02:14.704702 30697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "f919079462d04b38a282e3e919d43d24"
format_stamp: "Formatted at 2026-09-24 16:02:14 on dist-test-slave-n106"
server_key: "074011dbb7fb090ad708e570ea22287a"
server_key_iv: "b71cc5e630c14ce2e6998992af2bc549"
server_key_version: "encryptionkey@0"
I20260924 16:02:14.708341 30697 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:02:14.710731 30711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:14.711921 30697 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 16:02:14.712074 30697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "f919079462d04b38a282e3e919d43d24"
format_stamp: "Formatted at 2026-09-24 16:02:14 on dist-test-slave-n106"
server_key: "074011dbb7fb090ad708e570ea22287a"
server_key_iv: "b71cc5e630c14ce2e6998992af2bc549"
server_key_version: "encryptionkey@0"
I20260924 16:02:14.712191 30697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:14.716792 30697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:14.717391 30697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:14.717577 30697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:14.724737 30697 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:44639
I20260924 16:02:14.724761 30764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:44639 every 8 connection(s)
I20260924 16:02:14.725692 30697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:02:14.728217 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 30697
I20260924 16:02:14.728355 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:02:14.728667 26036 external_mini_cluster.cc:1428] Setting key 2d6a3bf19dd12320fd22cf5ac0080250
I20260924 16:02:14.730562 30765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:14.735003 30768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:14.730172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52936 (local address 127.25.109.62:44639)
0924 16:02:14.730909 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:14.730930 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:14.730975 (+ 45us) server_negotiation.cc:407] Connection header received
0924 16:02:14.731840 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:14.731864 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:14.732325 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:14.732717 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:14.733474 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:14.733480 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:14.733495 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:02:14.733507 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:14.733577 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:02:14.733599 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:14.733742 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:14.733816 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 16:02:14.734089 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":138,"threads_started":1}
I20260924 16:02:14.737735 30765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f919079462d04b38a282e3e919d43d24: Bootstrap starting.
I20260924 16:02:14.739907 30765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f919079462d04b38a282e3e919d43d24: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:14.740783 30765 log.cc:824] T 00000000000000000000000000000000 P f919079462d04b38a282e3e919d43d24: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:14.742733 30765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f919079462d04b38a282e3e919d43d24: No bootstrap required, opened a new log
I20260924 16:02:14.746506 30765 raft_consensus.cc:348] T 00000000000000000000000000000000 P f919079462d04b38a282e3e919d43d24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f919079462d04b38a282e3e919d43d24" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44639 } }
I20260924 16:02:14.746765 30765 raft_consensus.cc:374] T 00000000000000000000000000000000 P f919079462d04b38a282e3e919d43d24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:14.746863 30765 raft_consensus.cc:729] T 00000000000000000000000000000000 P f919079462d04b38a282e3e919d43d24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f919079462d04b38a282e3e919d43d24, State: Initialized, Role: FOLLOWER
I20260924 16:02:14.747272 30765 consensus_queue.cc:260] T 00000000000000000000000000000000 P f919079462d04b38a282e3e919d43d24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f919079462d04b38a282e3e919d43d24" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44639 } }
I20260924 16:02:14.747398 30765 raft_consensus.cc:388] T 00000000000000000000000000000000 P f919079462d04b38a282e3e919d43d24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:02:14.747447 30765 raft_consensus.cc:482] T 00000000000000000000000000000000 P f919079462d04b38a282e3e919d43d24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:02:14.747531 30765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f919079462d04b38a282e3e919d43d24 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:14.748740 30765 raft_consensus.cc:504] T 00000000000000000000000000000000 P f919079462d04b38a282e3e919d43d24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f919079462d04b38a282e3e919d43d24" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44639 } }
I20260924 16:02:14.749109 30765 leader_election.cc:302] T 00000000000000000000000000000000 P f919079462d04b38a282e3e919d43d24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f919079462d04b38a282e3e919d43d24; no voters:
I20260924 16:02:14.749468 30765 leader_election.cc:288] T 00000000000000000000000000000000 P f919079462d04b38a282e3e919d43d24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:02:14.749598 30770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f919079462d04b38a282e3e919d43d24 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:14.749872 30770 raft_consensus.cc:686] T 00000000000000000000000000000000 P f919079462d04b38a282e3e919d43d24 [term 1 LEADER]: Becoming Leader. State: Replica: f919079462d04b38a282e3e919d43d24, State: Running, Role: LEADER
I20260924 16:02:14.750201 30770 consensus_queue.cc:237] T 00000000000000000000000000000000 P f919079462d04b38a282e3e919d43d24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f919079462d04b38a282e3e919d43d24" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44639 } }
I20260924 16:02:14.750869 30765 sys_catalog.cc:564] T 00000000000000000000000000000000 P f919079462d04b38a282e3e919d43d24 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:02:14.752074 30771 sys_catalog.cc:455] T 00000000000000000000000000000000 P f919079462d04b38a282e3e919d43d24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f919079462d04b38a282e3e919d43d24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f919079462d04b38a282e3e919d43d24" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44639 } } }
I20260924 16:02:14.752209 30771 sys_catalog.cc:458] T 00000000000000000000000000000000 P f919079462d04b38a282e3e919d43d24 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:14.752465 30772 sys_catalog.cc:455] T 00000000000000000000000000000000 P f919079462d04b38a282e3e919d43d24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f919079462d04b38a282e3e919d43d24. Latest consensus state: current_term: 1 leader_uuid: "f919079462d04b38a282e3e919d43d24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f919079462d04b38a282e3e919d43d24" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44639 } } }
I20260924 16:02:14.752568 30772 sys_catalog.cc:458] T 00000000000000000000000000000000 P f919079462d04b38a282e3e919d43d24 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:14.752722 30779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:02:14.756114 30779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:02:14.762169 30779 catalog_manager.cc:1347] Generated new cluster ID: 434fb176363643eeb445de999ddb18ab
I20260924 16:02:14.762249 30779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:02:14.779201 30779 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:02:14.779618 30779 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:02:14.795032 30779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f919079462d04b38a282e3e919d43d24: Generated new TSK 0
I20260924 16:02:14.795825 30779 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:02:14.805004 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:0
--local_ip_for_outbound_sockets=127.25.109.1
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:44639
--builtin_ntp_servers=127.25.109.20:43079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 16:02:14.919333 30789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:14.919574 30789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:14.919613 30789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:14.923698 30789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:14.923753 30789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:14.923837 30789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:02:14.928594 30789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:43079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:44639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:14.929632 30789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:14.931046 30789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:14.940670 30794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:14.940724 30795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:14.940724 30789 server_base.cc:1034] running on GCE node
W20260924 16:02:14.940944 30797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:14.941313 30789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:14.941977 30789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:14.943209 30789 hybrid_clock.cc:648] HybridClock initialized: now 1790265734943176 us; error 44 us; skew 500 ppm
I20260924 16:02:14.945164 30789 webserver.cc:460] Webserver started at http://127.25.109.1:41085/ using document root <none> and password file <none>
I20260924 16:02:14.945752 30789 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:14.945847 30789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:14.946079 30789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:14.948105 30789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-0/data/instance:
uuid: "38aeaf67f74d47eb81a1b6eb6b028ce3"
format_stamp: "Formatted at 2026-09-24 16:02:14 on dist-test-slave-n106"
server_key: "6f0efcf4aa52b0ec51864dc31e78e0f4"
server_key_iv: "322c286114270cadda6b5498efe5d55f"
server_key_version: "encryptionkey@0"
I20260924 16:02:14.948715 30789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance:
uuid: "38aeaf67f74d47eb81a1b6eb6b028ce3"
format_stamp: "Formatted at 2026-09-24 16:02:14 on dist-test-slave-n106"
server_key: "6f0efcf4aa52b0ec51864dc31e78e0f4"
server_key_iv: "322c286114270cadda6b5498efe5d55f"
server_key_version: "encryptionkey@0"
I20260924 16:02:14.952576 30789 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:02:14.955144 30803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:14.956370 30789 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:02:14.956545 30789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "38aeaf67f74d47eb81a1b6eb6b028ce3"
format_stamp: "Formatted at 2026-09-24 16:02:14 on dist-test-slave-n106"
server_key: "6f0efcf4aa52b0ec51864dc31e78e0f4"
server_key_iv: "322c286114270cadda6b5498efe5d55f"
server_key_version: "encryptionkey@0"
I20260924 16:02:14.956676 30789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:14.961148 30789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:14.961740 30789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:14.961915 30789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:14.962435 30789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:14.963333 30789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:14.963405 30789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:14.963510 30789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:14.963560 30789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:14.973304 30789 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:36285
I20260924 16:02:14.973327 30917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:36285 every 8 connection(s)
I20260924 16:02:14.974231 30789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:02:14.980823 30768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:14.976171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:40277 (local address 127.25.109.62:44639)
0924 16:02:14.976348 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:14.976353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:14.977472 (+ 1119us) server_negotiation.cc:407] Connection header received
0924 16:02:14.978817 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:14.978820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:14.978859 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:14.978938 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:14.980298 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:14.980305 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:14.980309 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:02:14.980324 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:14.980351 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:02:14.980369 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:14.980521 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:14.980669 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 16:02:14.980711 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 16:02:14.981362 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 30789
I20260924 16:02:14.981515 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance
I20260924 16:02:14.981756 30920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:14.976615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:44639 (local address 127.25.109.1:40277)
0924 16:02:14.977301 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:14.977343 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:14.978581 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:14.979179 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:14.979187 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:14.979278 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:14.979837 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:02:14.979839 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:14.979897 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:02:14.980049 (+ 152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:14.980512 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:14.980520 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:14.980525 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 16:02:14.980684 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 16:02:14.980878 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":108,"threads_started":1}
I20260924 16:02:14.981845 26036 external_mini_cluster.cc:1428] Setting key 4524d6de80789ac67bac67e93452cade
I20260924 16:02:14.983152 30918 heartbeater.cc:347] Connected to a master server at 127.25.109.62:44639
I20260924 16:02:14.983385 30918 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:14.983799 30918 heartbeater.cc:511] Master 127.25.109.62:44639 requested a full tablet report, sending...
I20260924 16:02:14.984061 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:0
--local_ip_for_outbound_sockets=127.25.109.2
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:44639
--builtin_ntp_servers=127.25.109.20:43079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 16:02:14.985534 30728 ts_manager.cc:194] Registered new tserver with Master: 38aeaf67f74d47eb81a1b6eb6b028ce3 (127.25.109.1:36285)
W20260924 16:02:15.093698 30921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:15.093926 30921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:15.093964 30921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:15.097802 30921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:15.097846 30921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:15.097898 30921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:02:15.102725 30921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:43079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:44639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:15.103734 30921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:15.105031 30921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:15.114338 30928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:15.114344 30930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:15.114403 30927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:15.114449 30921 server_base.cc:1034] running on GCE node
I20260924 16:02:15.115056 30921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:15.115715 30921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:15.116873 30921 hybrid_clock.cc:648] HybridClock initialized: now 1790265735116855 us; error 27 us; skew 500 ppm
I20260924 16:02:15.118618 30921 webserver.cc:460] Webserver started at http://127.25.109.2:33299/ using document root <none> and password file <none>
I20260924 16:02:15.119177 30921 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:15.119238 30921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:15.119447 30921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:15.121591 30921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-1/data/instance:
uuid: "b7ce5d2300874a38a8b99332b49bc553"
format_stamp: "Formatted at 2026-09-24 16:02:15 on dist-test-slave-n106"
server_key: "b8697d51e07c0f21e6b3613c8815ea7f"
server_key_iv: "2000bc8714032caef6047b9a4a535969"
server_key_version: "encryptionkey@0"
I20260924 16:02:15.122131 30921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance:
uuid: "b7ce5d2300874a38a8b99332b49bc553"
format_stamp: "Formatted at 2026-09-24 16:02:15 on dist-test-slave-n106"
server_key: "b8697d51e07c0f21e6b3613c8815ea7f"
server_key_iv: "2000bc8714032caef6047b9a4a535969"
server_key_version: "encryptionkey@0"
I20260924 16:02:15.125725 30921 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:02:15.128181 30936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:15.129480 30921 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:02:15.129631 30921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "b7ce5d2300874a38a8b99332b49bc553"
format_stamp: "Formatted at 2026-09-24 16:02:15 on dist-test-slave-n106"
server_key: "b8697d51e07c0f21e6b3613c8815ea7f"
server_key_iv: "2000bc8714032caef6047b9a4a535969"
server_key_version: "encryptionkey@0"
I20260924 16:02:15.129742 30921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:15.134116 30921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:15.134703 30921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:15.134884 30921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:15.135411 30921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:15.136291 30921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:15.136360 30921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:15.136472 30921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:15.136549 30921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:15.144912 30921 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:41505
I20260924 16:02:15.145004 31050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:41505 every 8 connection(s)
I20260924 16:02:15.145864 30921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:02:15.150596 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 30921
I20260924 16:02:15.150750 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance
I20260924 16:02:15.151064 26036 external_mini_cluster.cc:1428] Setting key 9243577bca56250bcc994b16a23fc055
I20260924 16:02:15.152338 30768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:15.147729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:59995 (local address 127.25.109.62:44639)
0924 16:02:15.147887 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:15.147891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:15.148997 (+ 1106us) server_negotiation.cc:407] Connection header received
0924 16:02:15.150407 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:15.150410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:15.150459 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:15.150569 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:15.151824 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:15.151828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:15.151830 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:02:15.151843 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:15.151868 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:02:15.151916 (+ 48us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:15.152009 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:15.152157 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 16:02:15.152204 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 16:02:15.153155 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:0
--local_ip_for_outbound_sockets=127.25.109.3
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:44639
--builtin_ntp_servers=127.25.109.20:43079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 16:02:15.153460 31053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:15.148109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:44639 (local address 127.25.109.2:59995)
0924 16:02:15.148841 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:15.148883 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:15.150151 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:15.150775 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:15.150785 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:15.150914 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:15.151525 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:02:15.151526 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:15.151567 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:02:15.151638 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:15.151995 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:15.152003 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:15.152009 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 16:02:15.152200 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 16:02:15.152470 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":102,"threads_started":1}
I20260924 16:02:15.155534 31051 heartbeater.cc:347] Connected to a master server at 127.25.109.62:44639
I20260924 16:02:15.155787 31051 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:15.156177 31051 heartbeater.cc:511] Master 127.25.109.62:44639 requested a full tablet report, sending...
I20260924 16:02:15.157253 30728 ts_manager.cc:194] Registered new tserver with Master: b7ce5d2300874a38a8b99332b49bc553 (127.25.109.2:41505)
W20260924 16:02:15.267163 31054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:15.267465 31054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:15.267546 31054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:15.271652 31054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:15.271737 31054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:15.271827 31054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:02:15.276614 31054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:43079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:44639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:15.277689 31054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:15.279072 31054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:15.288714 31060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:15.288724 31059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:15.288721 31062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:15.288932 31054 server_base.cc:1034] running on GCE node
I20260924 16:02:15.289376 31054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:15.289975 31054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:15.291157 31054 hybrid_clock.cc:648] HybridClock initialized: now 1790265735291138 us; error 35 us; skew 500 ppm
I20260924 16:02:15.293044 31054 webserver.cc:460] Webserver started at http://127.25.109.3:34215/ using document root <none> and password file <none>
I20260924 16:02:15.293613 31054 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:15.293700 31054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:15.293931 31054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:15.295908 31054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-2/data/instance:
uuid: "a66ca3d30a7143d78851e6c9914171b2"
format_stamp: "Formatted at 2026-09-24 16:02:15 on dist-test-slave-n106"
server_key: "f5f9075fd2f6b4438ba8700486445bfd"
server_key_iv: "4731b531d34f98326f2b2e028c9ede20"
server_key_version: "encryptionkey@0"
I20260924 16:02:15.296527 31054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance:
uuid: "a66ca3d30a7143d78851e6c9914171b2"
format_stamp: "Formatted at 2026-09-24 16:02:15 on dist-test-slave-n106"
server_key: "f5f9075fd2f6b4438ba8700486445bfd"
server_key_iv: "4731b531d34f98326f2b2e028c9ede20"
server_key_version: "encryptionkey@0"
I20260924 16:02:15.299997 31054 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:02:15.302470 31068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:15.303691 31054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 16:02:15.303834 31054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "a66ca3d30a7143d78851e6c9914171b2"
format_stamp: "Formatted at 2026-09-24 16:02:15 on dist-test-slave-n106"
server_key: "f5f9075fd2f6b4438ba8700486445bfd"
server_key_iv: "4731b531d34f98326f2b2e028c9ede20"
server_key_version: "encryptionkey@0"
I20260924 16:02:15.303947 31054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:15.308316 31054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:15.308903 31054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:15.309084 31054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:15.309598 31054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:15.310487 31054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:15.310555 31054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:15.310624 31054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:15.310675 31054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:15.319170 31054 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:45971
I20260924 16:02:15.319238 31182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:45971 every 8 connection(s)
I20260924 16:02:15.320060 31054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:02:15.320544 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 31054
I20260924 16:02:15.320685 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance
I20260924 16:02:15.321014 26036 external_mini_cluster.cc:1428] Setting key dfd32d75f8dc9e69a1825a2eac6e71d7
I20260924 16:02:15.326681 30768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:15.322058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:37585 (local address 127.25.109.62:44639)
0924 16:02:15.322214 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:15.322221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:15.323592 (+ 1371us) server_negotiation.cc:407] Connection header received
0924 16:02:15.324965 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:15.324968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:15.325010 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:15.325088 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:15.326196 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:15.326200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:15.326203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:02:15.326215 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:15.326240 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:02:15.326257 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:15.326350 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:15.326502 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 16:02:15.326549 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 16:02:15.327487 31185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:15.322655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:44639 (local address 127.25.109.3:37585)
0924 16:02:15.323463 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:15.323503 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:15.324771 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:15.325245 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:15.325251 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:15.325335 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:15.325934 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:02:15.325936 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:15.325975 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:02:15.326044 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:15.326337 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:15.326346 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:15.326352 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 16:02:15.326530 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 16:02:15.326780 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":138,"threads_started":1}
I20260924 16:02:15.328565 31183 heartbeater.cc:347] Connected to a master server at 127.25.109.62:44639
I20260924 16:02:15.328794 31183 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:15.329195 31183 heartbeater.cc:511] Master 127.25.109.62:44639 requested a full tablet report, sending...
I20260924 16:02:15.330200 30728 ts_manager.cc:194] Registered new tserver with Master: a66ca3d30a7143d78851e6c9914171b2 (127.25.109.3:45971)
I20260924 16:02:15.334820 26036 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:02:15.338438 30768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:15.336779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52938 (local address 127.25.109.62:44639)
0924 16:02:15.336929 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:15.336933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:15.337105 (+ 172us) server_negotiation.cc:407] Connection header received
0924 16:02:15.337254 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:15.337257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:15.337298 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:15.337380 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:15.338016 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:15.338019 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:15.338022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:02:15.338033 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:15.338054 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:02:15.338063 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:15.338155 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:15.338272 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 16:02:15.338314 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 16:02:15.341295 30728 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:02:15.344260 30728 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:02:15.355974 31197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:15.353153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:41505 (local address 127.0.0.1:33964)
0924 16:02:15.353718 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:15.353734 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:15.353814 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:15.354505 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:15.354524 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:15.354603 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:15.355138 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:02:15.355140 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:15.355162 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:02:15.355232 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:15.355671 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:15.355683 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:15.355689 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 16:02:15.355797 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 16:02:15.355865 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":84,"threads_started":1}
I20260924 16:02:15.356557 31194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:15.352766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:36285 (local address 127.0.0.1:47860)
0924 16:02:15.353052 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:15.353094 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:15.353198 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:15.354764 (+ 1566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:15.354770 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:15.354790 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:15.355529 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:02:15.355531 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:15.355542 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:02:15.355573 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:15.356292 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:15.356295 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:15.356298 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 16:02:15.356384 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 16:02:15.356440 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260924 16:02:15.356827 31195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:15.352891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47860 (local address 127.25.109.1:36285)
0924 16:02:15.354344 (+ 1453us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:15.354355 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:15.354377 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:02:15.354434 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:15.354442 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:15.354600 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:15.355329 (+ 729us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:15.356002 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:15.356009 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:15.356022 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:02:15.356033 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:15.356170 (+ 137us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:02:15.356196 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:15.356319 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:15.356419 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 16:02:15.356492 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1359,"thread_start_us":192,"threads_started":1}
I20260924 16:02:15.356854 31198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:15.353587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48258 (local address 127.25.109.3:45971)
0924 16:02:15.354118 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:15.354129 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:15.354149 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:02:15.354303 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:15.354309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:15.354479 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:15.354602 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:15.355843 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:15.355850 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:15.355861 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:02:15.355874 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:15.355917 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:02:15.355948 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:15.356063 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:15.356631 (+ 568us) server_negotiation.cc:299] Negotiation successful
0924 16:02:15.356701 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":92,"threads_started":1}
I20260924 16:02:15.356910 31199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:15.353547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33964 (local address 127.25.109.2:41505)
0924 16:02:15.354016 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:15.354030 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:15.354056 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:02:15.354113 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:15.354119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:15.354344 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:15.354494 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:15.355388 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:15.355396 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:15.355410 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:02:15.355425 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:15.355472 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:02:15.355518 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:15.355629 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:15.356174 (+ 545us) server_negotiation.cc:299] Negotiation successful
0924 16:02:15.356769 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":119,"threads_started":1}
I20260924 16:02:15.357342 31196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:15.353447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:45971 (local address 127.0.0.1:48258)
0924 16:02:15.354085 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:15.354097 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:15.354167 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:15.355287 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:15.355293 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:15.355311 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:15.355690 (+ 379us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:02:15.355692 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:15.355700 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:02:15.355718 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:15.356057 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:15.356062 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:15.356401 (+ 339us) client_negotiation.cc:769] Sending connection context
0924 16:02:15.356502 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 16:02:15.356539 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":105,"threads_started":1}
I20260924 16:02:15.359656 30984 tablet_service.cc:1467] Processing CreateTablet for tablet 46441082d2114bd2a9f9f1513aa340c2 (DEFAULT_TABLE table=test-table [id=a8ec5c9fe32749f0bf4007cc5c1a30ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:15.359759 31117 tablet_service.cc:1467] Processing CreateTablet for tablet 46441082d2114bd2a9f9f1513aa340c2 (DEFAULT_TABLE table=test-table [id=a8ec5c9fe32749f0bf4007cc5c1a30ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:15.359760 30851 tablet_service.cc:1467] Processing CreateTablet for tablet 46441082d2114bd2a9f9f1513aa340c2 (DEFAULT_TABLE table=test-table [id=a8ec5c9fe32749f0bf4007cc5c1a30ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:15.360798 30984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46441082d2114bd2a9f9f1513aa340c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:15.360818 31117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46441082d2114bd2a9f9f1513aa340c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:15.360818 30851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46441082d2114bd2a9f9f1513aa340c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:15.366935 31200 tablet_bootstrap.cc:492] T 46441082d2114bd2a9f9f1513aa340c2 P b7ce5d2300874a38a8b99332b49bc553: Bootstrap starting.
I20260924 16:02:15.368229 31201 tablet_bootstrap.cc:492] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2: Bootstrap starting.
I20260924 16:02:15.368604 31202 tablet_bootstrap.cc:492] T 46441082d2114bd2a9f9f1513aa340c2 P 38aeaf67f74d47eb81a1b6eb6b028ce3: Bootstrap starting.
I20260924 16:02:15.369815 31200 tablet_bootstrap.cc:654] T 46441082d2114bd2a9f9f1513aa340c2 P b7ce5d2300874a38a8b99332b49bc553: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:15.370381 31202 tablet_bootstrap.cc:654] T 46441082d2114bd2a9f9f1513aa340c2 P 38aeaf67f74d47eb81a1b6eb6b028ce3: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:15.370412 31201 tablet_bootstrap.cc:654] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:15.370745 31200 log.cc:824] T 46441082d2114bd2a9f9f1513aa340c2 P b7ce5d2300874a38a8b99332b49bc553: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:15.371157 31202 log.cc:824] T 46441082d2114bd2a9f9f1513aa340c2 P 38aeaf67f74d47eb81a1b6eb6b028ce3: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:15.371207 31201 log.cc:824] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:15.372933 31200 tablet_bootstrap.cc:492] T 46441082d2114bd2a9f9f1513aa340c2 P b7ce5d2300874a38a8b99332b49bc553: No bootstrap required, opened a new log
I20260924 16:02:15.373137 31200 ts_tablet_manager.cc:1397] T 46441082d2114bd2a9f9f1513aa340c2 P b7ce5d2300874a38a8b99332b49bc553: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260924 16:02:15.373224 31202 tablet_bootstrap.cc:492] T 46441082d2114bd2a9f9f1513aa340c2 P 38aeaf67f74d47eb81a1b6eb6b028ce3: No bootstrap required, opened a new log
I20260924 16:02:15.373405 31202 ts_tablet_manager.cc:1397] T 46441082d2114bd2a9f9f1513aa340c2 P 38aeaf67f74d47eb81a1b6eb6b028ce3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 16:02:15.373591 31201 tablet_bootstrap.cc:492] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2: No bootstrap required, opened a new log
I20260924 16:02:15.373827 31201 ts_tablet_manager.cc:1397] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 16:02:15.377388 31201 raft_consensus.cc:348] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38aeaf67f74d47eb81a1b6eb6b028ce3" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36285 } } peers { permanent_uuid: "a66ca3d30a7143d78851e6c9914171b2" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 45971 } } peers { permanent_uuid: "b7ce5d2300874a38a8b99332b49bc553" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 41505 } }
I20260924 16:02:15.377646 31201 raft_consensus.cc:374] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:15.377708 31201 raft_consensus.cc:729] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a66ca3d30a7143d78851e6c9914171b2, State: Initialized, Role: FOLLOWER
I20260924 16:02:15.377895 31200 raft_consensus.cc:348] T 46441082d2114bd2a9f9f1513aa340c2 P b7ce5d2300874a38a8b99332b49bc553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38aeaf67f74d47eb81a1b6eb6b028ce3" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36285 } } peers { permanent_uuid: "a66ca3d30a7143d78851e6c9914171b2" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 45971 } } peers { permanent_uuid: "b7ce5d2300874a38a8b99332b49bc553" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 41505 } }
I20260924 16:02:15.377895 31202 raft_consensus.cc:348] T 46441082d2114bd2a9f9f1513aa340c2 P 38aeaf67f74d47eb81a1b6eb6b028ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38aeaf67f74d47eb81a1b6eb6b028ce3" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36285 } } peers { permanent_uuid: "a66ca3d30a7143d78851e6c9914171b2" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 45971 } } peers { permanent_uuid: "b7ce5d2300874a38a8b99332b49bc553" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 41505 } }
I20260924 16:02:15.378147 31200 raft_consensus.cc:374] T 46441082d2114bd2a9f9f1513aa340c2 P b7ce5d2300874a38a8b99332b49bc553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:15.378157 31202 raft_consensus.cc:374] T 46441082d2114bd2a9f9f1513aa340c2 P 38aeaf67f74d47eb81a1b6eb6b028ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:15.378217 31200 raft_consensus.cc:729] T 46441082d2114bd2a9f9f1513aa340c2 P b7ce5d2300874a38a8b99332b49bc553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7ce5d2300874a38a8b99332b49bc553, State: Initialized, Role: FOLLOWER
I20260924 16:02:15.378217 31202 raft_consensus.cc:729] T 46441082d2114bd2a9f9f1513aa340c2 P 38aeaf67f74d47eb81a1b6eb6b028ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38aeaf67f74d47eb81a1b6eb6b028ce3, State: Initialized, Role: FOLLOWER
I20260924 16:02:15.378204 31201 consensus_queue.cc:260] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38aeaf67f74d47eb81a1b6eb6b028ce3" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36285 } } peers { permanent_uuid: "a66ca3d30a7143d78851e6c9914171b2" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 45971 } } peers { permanent_uuid: "b7ce5d2300874a38a8b99332b49bc553" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 41505 } }
I20260924 16:02:15.378660 31200 consensus_queue.cc:260] T 46441082d2114bd2a9f9f1513aa340c2 P b7ce5d2300874a38a8b99332b49bc553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38aeaf67f74d47eb81a1b6eb6b028ce3" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36285 } } peers { permanent_uuid: "a66ca3d30a7143d78851e6c9914171b2" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 45971 } } peers { permanent_uuid: "b7ce5d2300874a38a8b99332b49bc553" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 41505 } }
I20260924 16:02:15.378656 31202 consensus_queue.cc:260] T 46441082d2114bd2a9f9f1513aa340c2 P 38aeaf67f74d47eb81a1b6eb6b028ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38aeaf67f74d47eb81a1b6eb6b028ce3" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36285 } } peers { permanent_uuid: "a66ca3d30a7143d78851e6c9914171b2" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 45971 } } peers { permanent_uuid: "b7ce5d2300874a38a8b99332b49bc553" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 41505 } }
I20260924 16:02:15.379271 31201 ts_tablet_manager.cc:1428] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 16:02:15.379325 31051 heartbeater.cc:503] Master 127.25.109.62:44639 was elected leader, sending a full tablet report...
I20260924 16:02:15.379582 31200 ts_tablet_manager.cc:1428] T 46441082d2114bd2a9f9f1513aa340c2 P b7ce5d2300874a38a8b99332b49bc553: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 16:02:15.379595 31183 heartbeater.cc:503] Master 127.25.109.62:44639 was elected leader, sending a full tablet report...
I20260924 16:02:15.380360 31202 ts_tablet_manager.cc:1428] T 46441082d2114bd2a9f9f1513aa340c2 P 38aeaf67f74d47eb81a1b6eb6b028ce3: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 16:02:15.381073 30918 heartbeater.cc:503] Master 127.25.109.62:44639 was elected leader, sending a full tablet report...
W20260924 16:02:15.396898 31052 tablet.cc:2369] T 46441082d2114bd2a9f9f1513aa340c2 P b7ce5d2300874a38a8b99332b49bc553: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:02:15.475417 30919 tablet.cc:2369] T 46441082d2114bd2a9f9f1513aa340c2 P 38aeaf67f74d47eb81a1b6eb6b028ce3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:02:15.519929 31206 raft_consensus.cc:482] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:02:15.520138 31206 raft_consensus.cc:504] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38aeaf67f74d47eb81a1b6eb6b028ce3" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36285 } } peers { permanent_uuid: "a66ca3d30a7143d78851e6c9914171b2" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 45971 } } peers { permanent_uuid: "b7ce5d2300874a38a8b99332b49bc553" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 41505 } }
I20260924 16:02:15.521366 31206 leader_election.cc:288] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38aeaf67f74d47eb81a1b6eb6b028ce3 (127.25.109.1:36285), b7ce5d2300874a38a8b99332b49bc553 (127.25.109.2:41505)
I20260924 16:02:15.523322 31185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:15.521585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:36285 (local address 127.25.109.3:45587)
0924 16:02:15.521715 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:15.521734 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:15.521825 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:15.522144 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:15.522148 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:15.522165 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:15.522666 (+ 501us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:02:15.522671 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:15.522685 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:02:15.522721 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:15.523024 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:15.523027 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:15.523031 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 16:02:15.523122 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 16:02:15.523157 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 16:02:15.523337 31195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:15.521701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:45587 (local address 127.25.109.1:36285)
0924 16:02:15.521867 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:15.521873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:15.521892 (+ 19us) server_negotiation.cc:407] Connection header received
0924 16:02:15.521947 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:15.521950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:15.521996 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:15.522123 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:15.522877 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:15.522880 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:15.522885 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:02:15.522903 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:15.522923 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:02:15.522931 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:15.523030 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:15.523141 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 16:02:15.523188 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 16:02:15.523532 31199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:15.521894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:36657 (local address 127.25.109.2:41505)
0924 16:02:15.522022 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:15.522026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:15.522127 (+ 101us) server_negotiation.cc:407] Connection header received
0924 16:02:15.522274 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:15.522277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:15.522311 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:15.522410 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:15.523108 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:15.523114 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:15.523117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:02:15.523129 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:15.523151 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:02:15.523161 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:15.523248 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:15.523343 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 16:02:15.523385 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 16:02:15.523871 31209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:15.521696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:41505 (local address 127.25.109.3:36657)
0924 16:02:15.522031 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:15.522043 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:15.522153 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:15.522462 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:15.522464 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:15.522477 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:15.522900 (+ 423us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:02:15.522902 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:15.522910 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:02:15.522935 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:15.523257 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:15.523260 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:15.523263 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 16:02:15.523350 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 16:02:15.523764 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":118,"threads_started":1}
I20260924 16:02:15.524094 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46441082d2114bd2a9f9f1513aa340c2" candidate_uuid: "a66ca3d30a7143d78851e6c9914171b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38aeaf67f74d47eb81a1b6eb6b028ce3" is_pre_election: true
I20260924 16:02:15.524487 30871 raft_consensus.cc:2454] T 46441082d2114bd2a9f9f1513aa340c2 P 38aeaf67f74d47eb81a1b6eb6b028ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a66ca3d30a7143d78851e6c9914171b2 in term 0.
I20260924 16:02:15.524560 31005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46441082d2114bd2a9f9f1513aa340c2" candidate_uuid: "a66ca3d30a7143d78851e6c9914171b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7ce5d2300874a38a8b99332b49bc553" is_pre_election: true
I20260924 16:02:15.524825 31005 raft_consensus.cc:2454] T 46441082d2114bd2a9f9f1513aa340c2 P b7ce5d2300874a38a8b99332b49bc553 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a66ca3d30a7143d78851e6c9914171b2 in term 0.
I20260924 16:02:15.524991 31069 leader_election.cc:302] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38aeaf67f74d47eb81a1b6eb6b028ce3, a66ca3d30a7143d78851e6c9914171b2; no voters:
I20260924 16:02:15.525238 31206 raft_consensus.cc:2781] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:02:15.525327 31206 raft_consensus.cc:482] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:02:15.525385 31206 raft_consensus.cc:3037] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:15.526432 31206 raft_consensus.cc:504] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38aeaf67f74d47eb81a1b6eb6b028ce3" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36285 } } peers { permanent_uuid: "a66ca3d30a7143d78851e6c9914171b2" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 45971 } } peers { permanent_uuid: "b7ce5d2300874a38a8b99332b49bc553" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 41505 } }
I20260924 16:02:15.526811 31206 leader_election.cc:288] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2 [CANDIDATE]: Term 1 election: Requested vote from peers 38aeaf67f74d47eb81a1b6eb6b028ce3 (127.25.109.1:36285), b7ce5d2300874a38a8b99332b49bc553 (127.25.109.2:41505)
I20260924 16:02:15.527158 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46441082d2114bd2a9f9f1513aa340c2" candidate_uuid: "a66ca3d30a7143d78851e6c9914171b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38aeaf67f74d47eb81a1b6eb6b028ce3"
I20260924 16:02:15.527258 31005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46441082d2114bd2a9f9f1513aa340c2" candidate_uuid: "a66ca3d30a7143d78851e6c9914171b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7ce5d2300874a38a8b99332b49bc553"
I20260924 16:02:15.527300 30871 raft_consensus.cc:3037] T 46441082d2114bd2a9f9f1513aa340c2 P 38aeaf67f74d47eb81a1b6eb6b028ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:15.527412 31005 raft_consensus.cc:3037] T 46441082d2114bd2a9f9f1513aa340c2 P b7ce5d2300874a38a8b99332b49bc553 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:15.528347 30871 raft_consensus.cc:2454] T 46441082d2114bd2a9f9f1513aa340c2 P 38aeaf67f74d47eb81a1b6eb6b028ce3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a66ca3d30a7143d78851e6c9914171b2 in term 1.
I20260924 16:02:15.528546 31005 raft_consensus.cc:2454] T 46441082d2114bd2a9f9f1513aa340c2 P b7ce5d2300874a38a8b99332b49bc553 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a66ca3d30a7143d78851e6c9914171b2 in term 1.
I20260924 16:02:15.528669 31069 leader_election.cc:302] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38aeaf67f74d47eb81a1b6eb6b028ce3, a66ca3d30a7143d78851e6c9914171b2; no voters:
I20260924 16:02:15.528846 31206 raft_consensus.cc:2781] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:15.529071 31206 raft_consensus.cc:686] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2 [term 1 LEADER]: Becoming Leader. State: Replica: a66ca3d30a7143d78851e6c9914171b2, State: Running, Role: LEADER
I20260924 16:02:15.529389 31206 consensus_queue.cc:237] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38aeaf67f74d47eb81a1b6eb6b028ce3" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36285 } } peers { permanent_uuid: "a66ca3d30a7143d78851e6c9914171b2" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 45971 } } peers { permanent_uuid: "b7ce5d2300874a38a8b99332b49bc553" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 41505 } }
I20260924 16:02:15.532701 30728 catalog_manager.cc:5495] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a66ca3d30a7143d78851e6c9914171b2 (127.25.109.3). New cstate: current_term: 1 leader_uuid: "a66ca3d30a7143d78851e6c9914171b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38aeaf67f74d47eb81a1b6eb6b028ce3" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a66ca3d30a7143d78851e6c9914171b2" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 45971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7ce5d2300874a38a8b99332b49bc553" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 41505 } health_report { overall_health: UNKNOWN } } }
I20260924 16:02:15.544780 31198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:15.543217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48268 (local address 127.25.109.3:45971)
0924 16:02:15.543344 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:15.543350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:15.543366 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:02:15.543601 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:15.543604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:15.543659 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:15.543750 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:15.544355 (+ 605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:15.544359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:15.544361 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:02:15.544372 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:15.544392 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:02:15.544440 (+ 48us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:15.544530 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:15.544594 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 16:02:15.544635 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 16:02:15.550364 30871 raft_consensus.cc:1261] T 46441082d2114bd2a9f9f1513aa340c2 P 38aeaf67f74d47eb81a1b6eb6b028ce3 [term 1 FOLLOWER]: Refusing update from remote peer a66ca3d30a7143d78851e6c9914171b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:02:15.550364 31005 raft_consensus.cc:1261] T 46441082d2114bd2a9f9f1513aa340c2 P b7ce5d2300874a38a8b99332b49bc553 [term 1 FOLLOWER]: Refusing update from remote peer a66ca3d30a7143d78851e6c9914171b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:02:15.551093 31210 consensus_queue.cc:1035] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38aeaf67f74d47eb81a1b6eb6b028ce3" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:15.551267 31206 consensus_queue.cc:1035] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7ce5d2300874a38a8b99332b49bc553" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 41505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:15.560765 31216 mvcc.cc:205] Tried to move back new op lower bound from 7332928452807573504 to 7332928452730925056. Current Snapshot: MvccSnapshot[applied={T|T < 7332928452807573504 or (T in {7332928452807573504})}]
I20260924 16:02:15.562484 31215 mvcc.cc:205] Tried to move back new op lower bound from 7332928452807573504 to 7332928452730925056. Current Snapshot: MvccSnapshot[applied={T|T < 7332928452807573504}]
I20260924 16:02:15.567906 30728 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:02:15.568089 30728 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:02:15.570585 30728 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f919079462d04b38a282e3e919d43d24: Sending DeleteTablet for 3 replicas of tablet 46441082d2114bd2a9f9f1513aa340c2
I20260924 16:02:15.571363 30984 tablet_service.cc:1514] Processing DeleteTablet for tablet 46441082d2114bd2a9f9f1513aa340c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:15 UTC) from {username='slave'} at 127.0.0.1:33964
I20260924 16:02:15.571417 30851 tablet_service.cc:1514] Processing DeleteTablet for tablet 46441082d2114bd2a9f9f1513aa340c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:15 UTC) from {username='slave'} at 127.0.0.1:47860
I20260924 16:02:15.571841 31225 tablet_replica.cc:330] T 46441082d2114bd2a9f9f1513aa340c2 P 38aeaf67f74d47eb81a1b6eb6b028ce3: stopping tablet replica
I20260924 16:02:15.571973 31117 tablet_service.cc:1514] Processing DeleteTablet for tablet 46441082d2114bd2a9f9f1513aa340c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:15 UTC) from {username='slave'} at 127.0.0.1:48258
I20260924 16:02:15.572260 31224 tablet_replica.cc:330] T 46441082d2114bd2a9f9f1513aa340c2 P b7ce5d2300874a38a8b99332b49bc553: stopping tablet replica
I20260924 16:02:15.572268 31225 raft_consensus.cc:2229] T 46441082d2114bd2a9f9f1513aa340c2 P 38aeaf67f74d47eb81a1b6eb6b028ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:15.572382 31226 tablet_replica.cc:330] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2: stopping tablet replica
I20260924 16:02:15.572660 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 30789
I20260924 16:02:15.572700 31226 raft_consensus.cc:2229] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:02:15.573040 31226 raft_consensus.cc:2258] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:15.572566 31224 raft_consensus.cc:2229] T 46441082d2114bd2a9f9f1513aa340c2 P b7ce5d2300874a38a8b99332b49bc553 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:15.573966 31224 raft_consensus.cc:2258] T 46441082d2114bd2a9f9f1513aa340c2 P b7ce5d2300874a38a8b99332b49bc553 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:15.575446 31224 ts_tablet_manager.cc:1905] T 46441082d2114bd2a9f9f1513aa340c2 P b7ce5d2300874a38a8b99332b49bc553: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:15.575446 31226 ts_tablet_manager.cc:1905] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:15.578210 31226 ts_tablet_manager.cc:1918] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:02:15.578210 31224 ts_tablet_manager.cc:1918] T 46441082d2114bd2a9f9f1513aa340c2 P b7ce5d2300874a38a8b99332b49bc553: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:02:15.578312 31224 log.cc:1196] T 46441082d2114bd2a9f9f1513aa340c2 P b7ce5d2300874a38a8b99332b49bc553: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-1/wal/wals/46441082d2114bd2a9f9f1513aa340c2
I20260924 16:02:15.578428 31226 log.cc:1196] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265708917108-26036-0/minicluster-data/ts-2/wal/wals/46441082d2114bd2a9f9f1513aa340c2
I20260924 16:02:15.578653 31224 ts_tablet_manager.cc:1939] T 46441082d2114bd2a9f9f1513aa340c2 P b7ce5d2300874a38a8b99332b49bc553: Deleting consensus metadata
I20260924 16:02:15.578891 31226 ts_tablet_manager.cc:1939] T 46441082d2114bd2a9f9f1513aa340c2 P a66ca3d30a7143d78851e6c9914171b2: Deleting consensus metadata
I20260924 16:02:15.579957 30714 catalog_manager.cc:4841] TS b7ce5d2300874a38a8b99332b49bc553 (127.25.109.2:41505): tablet 46441082d2114bd2a9f9f1513aa340c2 (table test-table [id=a8ec5c9fe32749f0bf4007cc5c1a30ee]) successfully deleted
I20260924 16:02:15.579957 30712 catalog_manager.cc:4841] TS a66ca3d30a7143d78851e6c9914171b2 (127.25.109.3:45971): tablet 46441082d2114bd2a9f9f1513aa340c2 (table test-table [id=a8ec5c9fe32749f0bf4007cc5c1a30ee]) successfully deleted
W20260924 16:02:15.581599 30712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.109.1:36285 (error 108)
I20260924 16:02:15.582415 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 30921
I20260924 16:02:15.582666 31196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:15.582353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:36285 (local address 127.0.0.1:47872)
0924 16:02:15.582518 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:15.582587 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.1:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 16:02:15.583376 30712 catalog_manager.cc:4584] TS 38aeaf67f74d47eb81a1b6eb6b028ce3 (127.25.109.1:36285): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 46441082d2114bd2a9f9f1513aa340c2: Network error: Client connection negotiation failed: client connection to 127.25.109.1:36285: connect: Connection refused (error 111)
I20260924 16:02:15.589679 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 31054
I20260924 16:02:15.594967 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 30697
2026-09-24T16:02:15Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1061 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 16:02:15.616299 26036 mini_oidc.cc:152] Starting JWKS server
I20260924 16:02:15.618666 26036 webserver.cc:460] Webserver started at https://127.0.0.1:39865/ using document root <none> and password file <none>
I20260924 16:02:15.618803 26036 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 16:02:15.619012 26036 webserver.cc:460] Webserver started at http://127.0.0.1:33275/ using document root <none> and password file <none>
2026-09-24T16:02:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:02:15Z Disabled control of system clock
I20260924 16:02:15.627941 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:39535
--webserver_interface=127.25.109.62
--webserver_port=0
--builtin_ntp_servers=127.25.109.20:34773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:39535
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39865/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265708917108-26036-0/testchainca.cert with env {}
W20260924 16:02:15.740581 31233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:15.740895 31233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:15.740937 31233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:15.745149 31233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:02:15.745216 31233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:15.745232 31233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:15.745250 31233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:02:15.745265 31233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 16:02:15.745282 31233 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 16:02:15.745296 31233 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39865/jwks.json
I20260924 16:02:15.750859 31233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:39535
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:39535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39865/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265708917108-26036-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:15.751989 31233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:15.753116 31233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:15.761735 31241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:15.761735 31238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:15.761806 31239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:15.761996 31233 server_base.cc:1034] running on GCE node
I20260924 16:02:15.762532 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:15.763799 31233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:15.764988 31233 hybrid_clock.cc:648] HybridClock initialized: now 1790265735764981 us; error 33 us; skew 500 ppm
I20260924 16:02:15.766921 31233 webserver.cc:460] Webserver started at http://127.25.109.62:35827/ using document root <none> and password file <none>
I20260924 16:02:15.767514 31233 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:15.767609 31233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:15.767853 31233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:15.770025 31233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "ad31a189637a4fdfb96bf5c79e601375"
format_stamp: "Formatted at 2026-09-24 16:02:15 on dist-test-slave-n106"
server_key: "e99d04b22a77bdceb957254a89cc5573"
server_key_iv: "2fa6a1f18cd1b1af462820c44180de9a"
server_key_version: "encryptionkey@0"
I20260924 16:02:15.770617 31233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "ad31a189637a4fdfb96bf5c79e601375"
format_stamp: "Formatted at 2026-09-24 16:02:15 on dist-test-slave-n106"
server_key: "e99d04b22a77bdceb957254a89cc5573"
server_key_iv: "2fa6a1f18cd1b1af462820c44180de9a"
server_key_version: "encryptionkey@0"
I20260924 16:02:15.774441 31233 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:02:15.776919 31247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:15.778347 31233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:02:15.778513 31233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "ad31a189637a4fdfb96bf5c79e601375"
format_stamp: "Formatted at 2026-09-24 16:02:15 on dist-test-slave-n106"
server_key: "e99d04b22a77bdceb957254a89cc5573"
server_key_iv: "2fa6a1f18cd1b1af462820c44180de9a"
server_key_version: "encryptionkey@0"
I20260924 16:02:15.778638 31233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:15.797544 31233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:15.798281 31233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:15.798487 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:15.805733 31233 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:39535
I20260924 16:02:15.805739 31299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:39535 every 8 connection(s)
I20260924 16:02:15.806684 31233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:02:15.810046 31300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:15.814203 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 31233
I20260924 16:02:15.814350 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:02:15.814613 26036 external_mini_cluster.cc:1428] Setting key c3b72e98005d97e4937d0f60a3e67f59
I20260924 16:02:15.815929 31300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad31a189637a4fdfb96bf5c79e601375: Bootstrap starting.
I20260924 16:02:15.818214 31300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad31a189637a4fdfb96bf5c79e601375: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:15.819022 31300 log.cc:824] T 00000000000000000000000000000000 P ad31a189637a4fdfb96bf5c79e601375: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:15.821130 31300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad31a189637a4fdfb96bf5c79e601375: No bootstrap required, opened a new log
I20260924 16:02:15.824540 31303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:15.815921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40956 (local address 127.25.109.62:39535)
0924 16:02:15.816645 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:15.816660 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:15.816698 (+ 38us) server_negotiation.cc:407] Connection header received
0924 16:02:15.817318 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:15.817334 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:15.817728 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:15.818070 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:15.818534 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:15.819585 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:15.820585 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:15.821263 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:15.821429 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:15.822211 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:15.822217 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:15.822233 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:02:15.822737 (+ 504us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:15.822806 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:02:15.822832 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:15.822931 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:15.823035 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 16:02:15.823322 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":138,"threads_started":1}
I20260924 16:02:15.825165 31300 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad31a189637a4fdfb96bf5c79e601375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad31a189637a4fdfb96bf5c79e601375" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 39535 } }
I20260924 16:02:15.825381 31300 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad31a189637a4fdfb96bf5c79e601375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:15.825443 31300 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad31a189637a4fdfb96bf5c79e601375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad31a189637a4fdfb96bf5c79e601375, State: Initialized, Role: FOLLOWER
I20260924 16:02:15.825893 31300 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad31a189637a4fdfb96bf5c79e601375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad31a189637a4fdfb96bf5c79e601375" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 39535 } }
I20260924 16:02:15.826009 31300 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad31a189637a4fdfb96bf5c79e601375 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:02:15.826049 31300 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad31a189637a4fdfb96bf5c79e601375 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:02:15.826122 31300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad31a189637a4fdfb96bf5c79e601375 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:15.827092 31300 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad31a189637a4fdfb96bf5c79e601375 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad31a189637a4fdfb96bf5c79e601375" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 39535 } }
I20260924 16:02:15.827316 26036 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:02:15.827499 31300 leader_election.cc:302] T 00000000000000000000000000000000 P ad31a189637a4fdfb96bf5c79e601375 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad31a189637a4fdfb96bf5c79e601375; no voters:
I20260924 16:02:15.827812 31300 leader_election.cc:288] T 00000000000000000000000000000000 P ad31a189637a4fdfb96bf5c79e601375 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:02:15.827937 31305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad31a189637a4fdfb96bf5c79e601375 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:15.828154 31305 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad31a189637a4fdfb96bf5c79e601375 [term 1 LEADER]: Becoming Leader. State: Replica: ad31a189637a4fdfb96bf5c79e601375, State: Running, Role: LEADER
I20260924 16:02:15.828537 31305 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad31a189637a4fdfb96bf5c79e601375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad31a189637a4fdfb96bf5c79e601375" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 39535 } }
I20260924 16:02:15.829192 31300 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad31a189637a4fdfb96bf5c79e601375 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:02:15.830315 31306 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad31a189637a4fdfb96bf5c79e601375 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad31a189637a4fdfb96bf5c79e601375. Latest consensus state: current_term: 1 leader_uuid: "ad31a189637a4fdfb96bf5c79e601375" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad31a189637a4fdfb96bf5c79e601375" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 39535 } } }
I20260924 16:02:15.830443 31306 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad31a189637a4fdfb96bf5c79e601375 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:15.830753 31307 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad31a189637a4fdfb96bf5c79e601375 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad31a189637a4fdfb96bf5c79e601375" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad31a189637a4fdfb96bf5c79e601375" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 39535 } } }
I20260924 16:02:15.830848 31314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:02:15.830863 31307 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad31a189637a4fdfb96bf5c79e601375 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:15.834187 31314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:02:15.839990 31314 catalog_manager.cc:1347] Generated new cluster ID: b172714d5b9f46c0b8df252a84643a31
I20260924 16:02:15.840056 31314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:02:15.865367 31314 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:02:15.866647 31314 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:02:15.876719 31314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad31a189637a4fdfb96bf5c79e601375: Generated new TSK 0
I20260924 16:02:15.877568 31314 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 16:02:15.906076 31330 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 16:02:15.906199 31330 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 16:02:15.906863 31303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:15.888874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40972 (local address 127.25.109.62:39535)
0924 16:02:15.889014 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:15.889019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:15.889207 (+ 188us) server_negotiation.cc:407] Connection header received
0924 16:02:15.889405 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:15.889408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:15.889469 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:15.889590 (+ 121us) server_negotiation.cc:226] Negotiated authn=JWT
0924 16:02:15.889829 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:15.890671 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:15.891511 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:15.891875 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:15.891977 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:15.892476 (+ 499us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 16:02:15.906416 (+ 13940us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39865/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 16:02:15.906719 (+ 303us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40972: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39865/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 16:02:15.906950 31303 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40972: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39865/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 16:02:15.908021 31329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:15.888747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39535 (local address 127.0.0.1:40972)
0924 16:02:15.889126 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:15.889140 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:15.889249 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:15.889606 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:15.889609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:15.889615 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0924 16:02:15.889724 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:15.889729 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:15.890785 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:15.890788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:15.891401 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:15.891408 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:15.892044 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:15.892051 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:15.892166 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:15.892233 (+ 67us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 16:02:15.906659 (+ 14426us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39865/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 16:02:15.906866 (+ 207us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.25.109.62:39535: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39865/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":108,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.25.109.62:39535: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39865/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 16:02:15.908804 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 31233
2026-09-24T16:02:15Z chronyd exiting
I20260924 16:02:15.946501 26036 test_util.cc:175] -----------------------------------------------
I20260924 16:02:15.946586 26036 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265708917108-26036-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (331 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 16:02:15.947671 26036 mini_oidc.cc:152] Starting JWKS server
I20260924 16:02:15.948558 26036 webserver.cc:460] Webserver started at https://127.0.0.1:40619/ using document root <none> and password file <none>
I20260924 16:02:15.948653 26036 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 16:02:15.948872 26036 webserver.cc:460] Webserver started at http://127.0.0.1:46381/ using document root <none> and password file <none>
2026-09-24T16:02:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:02:15Z Disabled control of system clock
I20260924 16:02:15.957841 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:41189
--webserver_interface=127.25.109.62
--webserver_port=0
--builtin_ntp_servers=127.25.109.20:36561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:41189
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40619/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265708917108-26036-0/testchainca.cert with env {}
W20260924 16:02:16.067373 31336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:16.067631 31336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:16.067668 31336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:16.071578 31336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:02:16.071632 31336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:16.071647 31336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:16.071664 31336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:02:16.071679 31336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 16:02:16.071694 31336 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 16:02:16.071708 31336 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40619/jwks.json
I20260924 16:02:16.077009 31336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:36561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:41189
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:41189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40619/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265708917108-26036-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:16.078011 31336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:16.078992 31336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:16.087752 31342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:16.087775 31336 server_base.cc:1034] running on GCE node
W20260924 16:02:16.087807 31344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:16.087780 31341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:16.088639 31336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:16.089745 31336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:16.090930 31336 hybrid_clock.cc:648] HybridClock initialized: now 1790265736090900 us; error 44 us; skew 500 ppm
I20260924 16:02:16.092890 31336 webserver.cc:460] Webserver started at http://127.25.109.62:42477/ using document root <none> and password file <none>
I20260924 16:02:16.093475 31336 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:16.093569 31336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:16.093807 31336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:16.095836 31336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "cadf5738dae74e40b216a1ad6e477009"
format_stamp: "Formatted at 2026-09-24 16:02:16 on dist-test-slave-n106"
server_key: "fd8eec9b9e5aa500e404ae472ff97511"
server_key_iv: "a6335631e8b70265709f08e414a52c4e"
server_key_version: "encryptionkey@0"
I20260924 16:02:16.096463 31336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "cadf5738dae74e40b216a1ad6e477009"
format_stamp: "Formatted at 2026-09-24 16:02:16 on dist-test-slave-n106"
server_key: "fd8eec9b9e5aa500e404ae472ff97511"
server_key_iv: "a6335631e8b70265709f08e414a52c4e"
server_key_version: "encryptionkey@0"
I20260924 16:02:16.100039 31336 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:02:16.102509 31351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:16.103753 31336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:02:16.103897 31336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "cadf5738dae74e40b216a1ad6e477009"
format_stamp: "Formatted at 2026-09-24 16:02:16 on dist-test-slave-n106"
server_key: "fd8eec9b9e5aa500e404ae472ff97511"
server_key_iv: "a6335631e8b70265709f08e414a52c4e"
server_key_version: "encryptionkey@0"
I20260924 16:02:16.104014 31336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:16.129715 31336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:16.130468 31336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:16.130681 31336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:16.137957 31336 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:41189
I20260924 16:02:16.137964 31404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:41189 every 8 connection(s)
I20260924 16:02:16.138944 31336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:02:16.142455 31405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:16.143550 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 31336
I20260924 16:02:16.143687 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:02:16.143981 26036 external_mini_cluster.cc:1428] Setting key d7a4c6b1b4708f2ace2e846d05d35f3b
I20260924 16:02:16.148588 31405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cadf5738dae74e40b216a1ad6e477009: Bootstrap starting.
I20260924 16:02:16.150889 31405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cadf5738dae74e40b216a1ad6e477009: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:16.151840 31405 log.cc:824] T 00000000000000000000000000000000 P cadf5738dae74e40b216a1ad6e477009: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:16.153556 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:16.145389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39870 (local address 127.25.109.62:41189)
0924 16:02:16.146064 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:16.146079 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:16.146125 (+ 46us) server_negotiation.cc:407] Connection header received
0924 16:02:16.146760 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:16.146777 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:16.147176 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:16.147515 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:16.147956 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:16.149133 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:16.150046 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:16.150538 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:16.150727 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:16.151648 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:16.151653 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:16.151667 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:02:16.152119 (+ 452us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:16.152183 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:02:16.152204 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:16.152330 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:16.152562 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 16:02:16.152838 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":147,"threads_started":1}
I20260924 16:02:16.154021 31405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cadf5738dae74e40b216a1ad6e477009: No bootstrap required, opened a new log
I20260924 16:02:16.156580 26036 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:02:16.158723 31405 raft_consensus.cc:348] T 00000000000000000000000000000000 P cadf5738dae74e40b216a1ad6e477009 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadf5738dae74e40b216a1ad6e477009" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41189 } }
I20260924 16:02:16.158969 31405 raft_consensus.cc:374] T 00000000000000000000000000000000 P cadf5738dae74e40b216a1ad6e477009 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:16.159031 31405 raft_consensus.cc:729] T 00000000000000000000000000000000 P cadf5738dae74e40b216a1ad6e477009 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cadf5738dae74e40b216a1ad6e477009, State: Initialized, Role: FOLLOWER
I20260924 16:02:16.159435 31405 consensus_queue.cc:260] T 00000000000000000000000000000000 P cadf5738dae74e40b216a1ad6e477009 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadf5738dae74e40b216a1ad6e477009" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41189 } }
I20260924 16:02:16.159591 31405 raft_consensus.cc:388] T 00000000000000000000000000000000 P cadf5738dae74e40b216a1ad6e477009 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:02:16.159677 31405 raft_consensus.cc:482] T 00000000000000000000000000000000 P cadf5738dae74e40b216a1ad6e477009 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:02:16.159776 31405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cadf5738dae74e40b216a1ad6e477009 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:16.160756 31405 raft_consensus.cc:504] T 00000000000000000000000000000000 P cadf5738dae74e40b216a1ad6e477009 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadf5738dae74e40b216a1ad6e477009" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41189 } }
I20260924 16:02:16.161099 31405 leader_election.cc:302] T 00000000000000000000000000000000 P cadf5738dae74e40b216a1ad6e477009 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cadf5738dae74e40b216a1ad6e477009; no voters:
I20260924 16:02:16.161434 31405 leader_election.cc:288] T 00000000000000000000000000000000 P cadf5738dae74e40b216a1ad6e477009 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:02:16.161542 31410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cadf5738dae74e40b216a1ad6e477009 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:16.161782 31410 raft_consensus.cc:686] T 00000000000000000000000000000000 P cadf5738dae74e40b216a1ad6e477009 [term 1 LEADER]: Becoming Leader. State: Replica: cadf5738dae74e40b216a1ad6e477009, State: Running, Role: LEADER
I20260924 16:02:16.162079 31410 consensus_queue.cc:237] T 00000000000000000000000000000000 P cadf5738dae74e40b216a1ad6e477009 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadf5738dae74e40b216a1ad6e477009" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41189 } }
I20260924 16:02:16.162726 31405 sys_catalog.cc:564] T 00000000000000000000000000000000 P cadf5738dae74e40b216a1ad6e477009 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:02:16.163719 31412 sys_catalog.cc:455] T 00000000000000000000000000000000 P cadf5738dae74e40b216a1ad6e477009 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cadf5738dae74e40b216a1ad6e477009. Latest consensus state: current_term: 1 leader_uuid: "cadf5738dae74e40b216a1ad6e477009" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadf5738dae74e40b216a1ad6e477009" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41189 } } }
I20260924 16:02:16.163872 31412 sys_catalog.cc:458] T 00000000000000000000000000000000 P cadf5738dae74e40b216a1ad6e477009 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:16.164137 31411 sys_catalog.cc:455] T 00000000000000000000000000000000 P cadf5738dae74e40b216a1ad6e477009 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cadf5738dae74e40b216a1ad6e477009" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadf5738dae74e40b216a1ad6e477009" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41189 } } }
I20260924 16:02:16.164227 31411 sys_catalog.cc:458] T 00000000000000000000000000000000 P cadf5738dae74e40b216a1ad6e477009 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:16.164230 31419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:02:16.167595 31419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:02:16.173599 31419 catalog_manager.cc:1347] Generated new cluster ID: e558c52f535745b5a69f769f47589a54
I20260924 16:02:16.173688 31419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:02:16.182734 31419 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:02:16.183658 31419 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:02:16.189874 31419 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cadf5738dae74e40b216a1ad6e477009: Generated new TSK 0
I20260924 16:02:16.190701 31419 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 16:02:16.219866 31435 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 16:02:16.219964 31435 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 16:02:16.220736 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:16.204706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39876 (local address 127.25.109.62:41189)
0924 16:02:16.204842 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:16.204847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:16.204999 (+ 152us) server_negotiation.cc:407] Connection header received
0924 16:02:16.205133 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:16.205136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:16.205192 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:16.205310 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0924 16:02:16.205517 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:16.206356 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:16.207285 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:16.207672 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:16.207782 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:16.208156 (+ 374us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 16:02:16.220265 (+ 12109us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40619/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 16:02:16.220586 (+ 321us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39876: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40619/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 16:02:16.220810 31408 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39876: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40619/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 16:02:16.220923 31434 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:16.204621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:41189 (local address 127.0.0.1:39876)
0924 16:02:16.204917 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:16.204929 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:16.205024 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:16.205312 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:16.205315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:16.205321 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0924 16:02:16.205414 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:16.205418 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:16.206475 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:16.206478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:16.207170 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:16.207179 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:16.207810 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:16.207814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:16.207900 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:16.207962 (+ 62us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 16:02:16.220556 (+ 12594us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40619/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 16:02:16.220721 (+ 165us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.25.109.62:41189: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40619/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":112,"threads_started":1}
I20260924 16:02:16.221679 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 31336
2026-09-24T16:02:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (314 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 16:02:16.261765 26036 mini_oidc.cc:152] Starting JWKS server
I20260924 16:02:16.262601 26036 webserver.cc:460] Webserver started at https://127.0.0.1:46169/ using document root <none> and password file <none>
I20260924 16:02:16.262684 26036 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 16:02:16.262889 26036 webserver.cc:460] Webserver started at http://127.0.0.1:38151/ using document root <none> and password file <none>
2026-09-24T16:02:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:02:16Z Disabled control of system clock
I20260924 16:02:16.271788 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:43789
--webserver_interface=127.25.109.62
--webserver_port=0
--builtin_ntp_servers=127.25.109.20:36097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:43789
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46169/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 16:02:16.388214 31441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:16.388567 31441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:16.388643 31441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:16.392815 31441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:02:16.392884 31441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:16.392900 31441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:16.392917 31441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:02:16.392968 31441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 16:02:16.393004 31441 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 16:02:16.393041 31441 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46169/jwks.json
I20260924 16:02:16.398144 31441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:36097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:43789
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:43789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46169/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:16.399320 31441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:16.400825 31441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:16.409631 31449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:16.409624 31446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:16.409631 31447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:16.409691 31441 server_base.cc:1034] running on GCE node
I20260924 16:02:16.410429 31441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:16.411602 31441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:16.412784 31441 hybrid_clock.cc:648] HybridClock initialized: now 1790265736412768 us; error 29 us; skew 500 ppm
I20260924 16:02:16.414677 31441 webserver.cc:460] Webserver started at http://127.25.109.62:39815/ using document root <none> and password file <none>
I20260924 16:02:16.415275 31441 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:16.415365 31441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:16.415602 31441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:16.417819 31441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "73d70b8b8c5c4efca9d58f3f151ce3d4"
format_stamp: "Formatted at 2026-09-24 16:02:16 on dist-test-slave-n106"
server_key: "2be2c095a563099fc9180ffbbc41443b"
server_key_iv: "6d171b87f907d861b0772d535ff02516"
server_key_version: "encryptionkey@0"
I20260924 16:02:16.418409 31441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "73d70b8b8c5c4efca9d58f3f151ce3d4"
format_stamp: "Formatted at 2026-09-24 16:02:16 on dist-test-slave-n106"
server_key: "2be2c095a563099fc9180ffbbc41443b"
server_key_iv: "6d171b87f907d861b0772d535ff02516"
server_key_version: "encryptionkey@0"
I20260924 16:02:16.423246 31441 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 16:02:16.426667 31455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:16.428355 31441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:02:16.428529 31441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "73d70b8b8c5c4efca9d58f3f151ce3d4"
format_stamp: "Formatted at 2026-09-24 16:02:16 on dist-test-slave-n106"
server_key: "2be2c095a563099fc9180ffbbc41443b"
server_key_iv: "6d171b87f907d861b0772d535ff02516"
server_key_version: "encryptionkey@0"
I20260924 16:02:16.428658 31441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:16.447683 31441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:16.448378 31441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:16.448644 31441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:16.455585 31441 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:43789
I20260924 16:02:16.455588 31508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:43789 every 8 connection(s)
I20260924 16:02:16.456666 31441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:02:16.457513 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 31441
I20260924 16:02:16.457667 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:02:16.457963 26036 external_mini_cluster.cc:1428] Setting key 01c8eabf8f4923b5e33225d1966b6e11
I20260924 16:02:16.461512 31509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:16.467746 31509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73d70b8b8c5c4efca9d58f3f151ce3d4: Bootstrap starting.
I20260924 16:02:16.468521 31512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:16.459444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40238 (local address 127.25.109.62:43789)
0924 16:02:16.460211 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:16.460228 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:16.460270 (+ 42us) server_negotiation.cc:407] Connection header received
0924 16:02:16.461093 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:16.461112 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:16.461565 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:16.462005 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:16.462510 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:16.463730 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:16.464842 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:16.465377 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:16.465583 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:16.466605 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:16.466610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:16.466624 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:02:16.467011 (+ 387us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:16.467071 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:02:16.467090 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:16.467205 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:16.467371 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 16:02:16.467661 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":166,"threads_started":1}
I20260924 16:02:16.470620 31509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73d70b8b8c5c4efca9d58f3f151ce3d4: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:16.471560 31509 log.cc:824] T 00000000000000000000000000000000 P 73d70b8b8c5c4efca9d58f3f151ce3d4: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:16.471951 26036 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:02:16.474207 31509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73d70b8b8c5c4efca9d58f3f151ce3d4: No bootstrap required, opened a new log
I20260924 16:02:16.478180 31509 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73d70b8b8c5c4efca9d58f3f151ce3d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73d70b8b8c5c4efca9d58f3f151ce3d4" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 43789 } }
I20260924 16:02:16.478422 31509 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73d70b8b8c5c4efca9d58f3f151ce3d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:16.478495 31509 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73d70b8b8c5c4efca9d58f3f151ce3d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73d70b8b8c5c4efca9d58f3f151ce3d4, State: Initialized, Role: FOLLOWER
I20260924 16:02:16.478994 31509 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73d70b8b8c5c4efca9d58f3f151ce3d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73d70b8b8c5c4efca9d58f3f151ce3d4" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 43789 } }
I20260924 16:02:16.479192 31509 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73d70b8b8c5c4efca9d58f3f151ce3d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:02:16.479281 31509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73d70b8b8c5c4efca9d58f3f151ce3d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:02:16.479408 31509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73d70b8b8c5c4efca9d58f3f151ce3d4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:16.480535 31509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73d70b8b8c5c4efca9d58f3f151ce3d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73d70b8b8c5c4efca9d58f3f151ce3d4" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 43789 } }
I20260924 16:02:16.480933 31509 leader_election.cc:302] T 00000000000000000000000000000000 P 73d70b8b8c5c4efca9d58f3f151ce3d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73d70b8b8c5c4efca9d58f3f151ce3d4; no voters:
I20260924 16:02:16.481284 31509 leader_election.cc:288] T 00000000000000000000000000000000 P 73d70b8b8c5c4efca9d58f3f151ce3d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:02:16.481415 31514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73d70b8b8c5c4efca9d58f3f151ce3d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:16.481696 31514 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73d70b8b8c5c4efca9d58f3f151ce3d4 [term 1 LEADER]: Becoming Leader. State: Replica: 73d70b8b8c5c4efca9d58f3f151ce3d4, State: Running, Role: LEADER
I20260924 16:02:16.482000 31514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73d70b8b8c5c4efca9d58f3f151ce3d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73d70b8b8c5c4efca9d58f3f151ce3d4" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 43789 } }
I20260924 16:02:16.482852 31509 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73d70b8b8c5c4efca9d58f3f151ce3d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:02:16.483938 31515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73d70b8b8c5c4efca9d58f3f151ce3d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73d70b8b8c5c4efca9d58f3f151ce3d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73d70b8b8c5c4efca9d58f3f151ce3d4" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 43789 } } }
I20260924 16:02:16.483906 31516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73d70b8b8c5c4efca9d58f3f151ce3d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73d70b8b8c5c4efca9d58f3f151ce3d4. Latest consensus state: current_term: 1 leader_uuid: "73d70b8b8c5c4efca9d58f3f151ce3d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73d70b8b8c5c4efca9d58f3f151ce3d4" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 43789 } } }
I20260924 16:02:16.484061 31515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73d70b8b8c5c4efca9d58f3f151ce3d4 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:16.484133 31516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73d70b8b8c5c4efca9d58f3f151ce3d4 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:16.484479 31523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:02:16.487749 31523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:02:16.493654 31523 catalog_manager.cc:1347] Generated new cluster ID: e8309228f60c46e5b2a3e10438cf9c96
I20260924 16:02:16.493796 31523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:02:16.509655 31523 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:02:16.510609 31523 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:02:16.516762 31523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73d70b8b8c5c4efca9d58f3f151ce3d4: Generated new TSK 0
I20260924 16:02:16.517447 31523 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:02:16.536445 31512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:16.531478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40254 (local address 127.25.109.62:43789)
0924 16:02:16.531608 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:16.531615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:16.531818 (+ 203us) server_negotiation.cc:407] Connection header received
0924 16:02:16.532014 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:16.532017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:16.532069 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:16.532183 (+ 114us) server_negotiation.cc:226] Negotiated authn=JWT
0924 16:02:16.532444 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:16.533073 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:16.534022 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:16.534253 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:16.534367 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:16.534680 (+ 313us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 16:02:16.536061 (+ 1381us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46169/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 16:02:16.536288 (+ 227us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40254: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46169/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 16:02:16.536516 31512 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40254: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46169/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 16:02:16.538309 31538 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:16.531390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:43789 (local address 127.0.0.1:40254)
0924 16:02:16.531729 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:16.531740 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:16.531871 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:16.532182 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:16.532185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:16.532192 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0924 16:02:16.532314 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:16.532319 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:16.533190 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:16.533194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:16.533897 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:16.533913 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:16.534378 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:16.534382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:16.534467 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:16.534499 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 16:02:16.536234 (+ 1735us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46169/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 16:02:16.538125 (+ 1891us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.25.109.62:43789: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46169/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":1682,"thread_start_us":118,"threads_started":1}
I20260924 16:02:16.538863 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 31441
E20260924 16:02:16.549839 31539 webserver.cc:533] Webserver: sslize failed:
E20260924 16:02:16.549911 31539 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T16:02:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (317 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T16:02:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:02:16Z Disabled control of system clock
I20260924 16:02:16.590080 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:45437
--webserver_interface=127.25.109.62
--webserver_port=0
--builtin_ntp_servers=127.25.109.20:37043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:45437
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265708917108-26036-0/testchainca.cert with env {}
W20260924 16:02:16.701023 31543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:16.701351 31543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:16.701429 31543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:16.705456 31543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:02:16.705547 31543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:16.705576 31543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:16.705612 31543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:02:16.705644 31543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 16:02:16.705680 31543 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 16:02:16.705713 31543 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 16:02:16.711033 31543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:37043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:45437
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:45437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265708917108-26036-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:16.712157 31543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:16.713238 31543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:16.721823 31548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:16.721788 31551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:16.721796 31549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:16.722154 31543 server_base.cc:1034] running on GCE node
I20260924 16:02:16.722708 31543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:16.723802 31543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:16.724959 31543 hybrid_clock.cc:648] HybridClock initialized: now 1790265736724926 us; error 47 us; skew 500 ppm
I20260924 16:02:16.726914 31543 webserver.cc:460] Webserver started at http://127.25.109.62:38521/ using document root <none> and password file <none>
I20260924 16:02:16.727519 31543 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:16.727582 31543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:16.727851 31543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:16.730007 31543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "4f8a6f1e735a4bfca8ad254e57f77b32"
format_stamp: "Formatted at 2026-09-24 16:02:16 on dist-test-slave-n106"
server_key: "4a231098175c383984737e49608a470e"
server_key_iv: "8efedf97911fb745a056e4870333e567"
server_key_version: "encryptionkey@0"
I20260924 16:02:16.730614 31543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "4f8a6f1e735a4bfca8ad254e57f77b32"
format_stamp: "Formatted at 2026-09-24 16:02:16 on dist-test-slave-n106"
server_key: "4a231098175c383984737e49608a470e"
server_key_iv: "8efedf97911fb745a056e4870333e567"
server_key_version: "encryptionkey@0"
I20260924 16:02:16.734436 31543 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260924 16:02:16.737097 31557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:16.738474 31543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:02:16.738643 31543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "4f8a6f1e735a4bfca8ad254e57f77b32"
format_stamp: "Formatted at 2026-09-24 16:02:16 on dist-test-slave-n106"
server_key: "4a231098175c383984737e49608a470e"
server_key_iv: "8efedf97911fb745a056e4870333e567"
server_key_version: "encryptionkey@0"
I20260924 16:02:16.738773 31543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:16.750182 31543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:16.750895 31543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:16.751104 31543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:16.758528 31543 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:45437
I20260924 16:02:16.758550 31609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:45437 every 8 connection(s)
I20260924 16:02:16.759491 31543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:02:16.762403 31610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:16.765635 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 31543
I20260924 16:02:16.765769 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:02:16.766096 26036 external_mini_cluster.cc:1428] Setting key 60093ab23d761213ae5954634aa06d24
I20260924 16:02:16.768515 31610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f8a6f1e735a4bfca8ad254e57f77b32: Bootstrap starting.
I20260924 16:02:16.770645 31610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f8a6f1e735a4bfca8ad254e57f77b32: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:16.771394 31610 log.cc:824] T 00000000000000000000000000000000 P 4f8a6f1e735a4bfca8ad254e57f77b32: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:16.773307 31610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f8a6f1e735a4bfca8ad254e57f77b32: No bootstrap required, opened a new log
I20260924 16:02:16.775816 31613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:16.767493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51882 (local address 127.25.109.62:45437)
0924 16:02:16.768141 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:16.768156 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:16.768191 (+ 35us) server_negotiation.cc:407] Connection header received
0924 16:02:16.768922 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:16.768941 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:16.769337 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:16.769679 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:16.770103 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:16.771158 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:16.772282 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:16.772821 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:16.773006 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:16.773908 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:16.773922 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:16.773935 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:02:16.774312 (+ 377us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:16.774367 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:02:16.774387 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:16.774509 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:16.774627 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 16:02:16.774878 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":127,"threads_started":1}
I20260924 16:02:16.776393 31610 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f8a6f1e735a4bfca8ad254e57f77b32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8a6f1e735a4bfca8ad254e57f77b32" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 45437 } }
I20260924 16:02:16.776671 31610 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f8a6f1e735a4bfca8ad254e57f77b32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:16.776726 31610 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f8a6f1e735a4bfca8ad254e57f77b32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8a6f1e735a4bfca8ad254e57f77b32, State: Initialized, Role: FOLLOWER
I20260924 16:02:16.777866 31610 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f8a6f1e735a4bfca8ad254e57f77b32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8a6f1e735a4bfca8ad254e57f77b32" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 45437 } }
I20260924 16:02:16.778002 31610 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f8a6f1e735a4bfca8ad254e57f77b32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:02:16.778095 31610 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f8a6f1e735a4bfca8ad254e57f77b32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:02:16.778211 31610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f8a6f1e735a4bfca8ad254e57f77b32 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:16.778715 26036 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:02:16.779163 31610 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f8a6f1e735a4bfca8ad254e57f77b32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8a6f1e735a4bfca8ad254e57f77b32" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 45437 } }
I20260924 16:02:16.779484 31610 leader_election.cc:302] T 00000000000000000000000000000000 P 4f8a6f1e735a4bfca8ad254e57f77b32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f8a6f1e735a4bfca8ad254e57f77b32; no voters:
I20260924 16:02:16.779827 31610 leader_election.cc:288] T 00000000000000000000000000000000 P 4f8a6f1e735a4bfca8ad254e57f77b32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:02:16.779935 31615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f8a6f1e735a4bfca8ad254e57f77b32 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:16.780169 31615 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f8a6f1e735a4bfca8ad254e57f77b32 [term 1 LEADER]: Becoming Leader. State: Replica: 4f8a6f1e735a4bfca8ad254e57f77b32, State: Running, Role: LEADER
I20260924 16:02:16.780516 31615 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f8a6f1e735a4bfca8ad254e57f77b32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8a6f1e735a4bfca8ad254e57f77b32" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 45437 } }
I20260924 16:02:16.781059 31610 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f8a6f1e735a4bfca8ad254e57f77b32 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:02:16.782275 31616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f8a6f1e735a4bfca8ad254e57f77b32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f8a6f1e735a4bfca8ad254e57f77b32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8a6f1e735a4bfca8ad254e57f77b32" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 45437 } } }
I20260924 16:02:16.782284 31617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f8a6f1e735a4bfca8ad254e57f77b32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f8a6f1e735a4bfca8ad254e57f77b32. Latest consensus state: current_term: 1 leader_uuid: "4f8a6f1e735a4bfca8ad254e57f77b32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8a6f1e735a4bfca8ad254e57f77b32" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 45437 } } }
I20260924 16:02:16.782405 31616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f8a6f1e735a4bfca8ad254e57f77b32 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:16.782414 31617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f8a6f1e735a4bfca8ad254e57f77b32 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:16.782814 31625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:02:16.786039 31625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:02:16.791651 31625 catalog_manager.cc:1347] Generated new cluster ID: 5f6aabc6eb43488e815799e331eed431
I20260924 16:02:16.791724 31625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:02:16.820582 31625 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:02:16.821722 31625 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:02:16.828724 31625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f8a6f1e735a4bfca8ad254e57f77b32: Generated new TSK 0
I20260924 16:02:16.829481 31625 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:02:16.889055 31613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:16.837553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51884 (local address 127.25.109.62:45437)
0924 16:02:16.837693 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:16.837698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:16.837834 (+ 136us) server_negotiation.cc:407] Connection header received
0924 16:02:16.837985 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:16.837989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:16.838049 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:16.838176 (+ 127us) server_negotiation.cc:226] Negotiated authn=JWT
0924 16:02:16.838386 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:16.839226 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:16.840083 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:16.840473 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:16.840581 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:16.840881 (+ 300us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 16:02:16.888537 (+ 47656us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 16:02:16.888891 (+ 354us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51884: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 16:02:16.889150 31613 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51884: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 16:02:16.889212 31639 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:16.837444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:45437 (local address 127.0.0.1:51884)
0924 16:02:16.837742 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:16.837755 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:16.837854 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:16.838177 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:16.838179 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:16.838185 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0924 16:02:16.838278 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:16.838282 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:16.839333 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:16.839335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:16.839952 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:16.839959 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:16.840594 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:16.840597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:16.840670 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:16.840694 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 16:02:16.888806 (+ 48112us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 16:02:16.888997 (+ 191us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.25.109.62:45437: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":119,"threads_started":1}
I20260924 16:02:16.889883 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 31543
2026-09-24T16:02:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (329 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (126 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (122 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:02:17 dist-test-slave-n106 krb5kdc[31651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:02:17 dist-test-slave-n106 krb5kdc[31651](info): set up 2 sockets
Sep 24 16:02:17 dist-test-slave-n106 krb5kdc[31651](info): commencing operation
krb5kdc: starting...
W20260924 16:02:18.185966 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:02:18 dist-test-slave-n106 krb5kdc[31651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:02:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:02:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
I20260924 16:02:18.305682 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:37739
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:37739
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:18.424132 31664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:18.424584 31664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:18.424669 31664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:18.429540 31664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:02:18.429605 31664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:18.429626 31664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:18.429643 31664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:02:18.429658 31664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:02:18.435091 31664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:37739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:37739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:18.436139 31664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:18.437234 31664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:18.445958 31669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:18.445961 31670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:18.445961 31672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:18.446079 31664 server_base.cc:1034] running on GCE node
I20260924 16:02:18.446889 31664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:18.448029 31664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:18.449190 31664 hybrid_clock.cc:648] HybridClock initialized: now 1790265738449164 us; error 38 us; skew 500 ppm
Sep 24 16:02:18 dist-test-slave-n106 krb5kdc[31651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265738, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:18.452157 31664 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:02:18.453155 31664 webserver.cc:460] Webserver started at http://127.25.109.62:44787/ using document root <none> and password file <none>
I20260924 16:02:18.453760 31664 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:18.453855 31664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:18.454105 31664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:18.456262 31664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "42171ced686844f4a7fd3f6a30f04e6b"
format_stamp: "Formatted at 2026-09-24 16:02:18 on dist-test-slave-n106"
server_key: "7f959b67cdf992aa5dff3f67ef63f0cb"
server_key_iv: "fa104aeccee5a8d738c2f10cdbcd089e"
server_key_version: "encryptionkey@0"
I20260924 16:02:18.456931 31664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "42171ced686844f4a7fd3f6a30f04e6b"
format_stamp: "Formatted at 2026-09-24 16:02:18 on dist-test-slave-n106"
server_key: "7f959b67cdf992aa5dff3f67ef63f0cb"
server_key_iv: "fa104aeccee5a8d738c2f10cdbcd089e"
server_key_version: "encryptionkey@0"
I20260924 16:02:18.460897 31664 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:02:18.463470 31679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:18.464782 31664 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:02:18.464928 31664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "42171ced686844f4a7fd3f6a30f04e6b"
format_stamp: "Formatted at 2026-09-24 16:02:18 on dist-test-slave-n106"
server_key: "7f959b67cdf992aa5dff3f67ef63f0cb"
server_key_iv: "fa104aeccee5a8d738c2f10cdbcd089e"
server_key_version: "encryptionkey@0"
I20260924 16:02:18.465049 31664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:18.481108 31664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:18.484516 31664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:18.484719 31664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:18.492056 31664 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:37739
I20260924 16:02:18.492074 31732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:37739 every 8 connection(s)
I20260924 16:02:18.493108 31664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:02:18.495818 31733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:18.501410 31733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42171ced686844f4a7fd3f6a30f04e6b: Bootstrap starting.
I20260924 16:02:18.502393 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 31664
I20260924 16:02:18.502525 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:02:18.502853 26036 external_mini_cluster.cc:1428] Setting key 55bfb14de7d3b88077d5154dc549dae1
I20260924 16:02:18.504458 31733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42171ced686844f4a7fd3f6a30f04e6b: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:18.505216 31733 log.cc:824] T 00000000000000000000000000000000 P 42171ced686844f4a7fd3f6a30f04e6b: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:18.507037 31733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42171ced686844f4a7fd3f6a30f04e6b: No bootstrap required, opened a new log
I20260924 16:02:18.510388 31733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42171ced686844f4a7fd3f6a30f04e6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42171ced686844f4a7fd3f6a30f04e6b" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 37739 } }
I20260924 16:02:18.510629 31733 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42171ced686844f4a7fd3f6a30f04e6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:18.510718 31733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42171ced686844f4a7fd3f6a30f04e6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42171ced686844f4a7fd3f6a30f04e6b, State: Initialized, Role: FOLLOWER
I20260924 16:02:18.511279 31733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42171ced686844f4a7fd3f6a30f04e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42171ced686844f4a7fd3f6a30f04e6b" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 37739 } }
I20260924 16:02:18.511454 31733 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42171ced686844f4a7fd3f6a30f04e6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:02:18.511523 31733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42171ced686844f4a7fd3f6a30f04e6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Sep 24 16:02:18 dist-test-slave-n106 krb5kdc[31651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:02:18.511626 31733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42171ced686844f4a7fd3f6a30f04e6b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:18.512553 31733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42171ced686844f4a7fd3f6a30f04e6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42171ced686844f4a7fd3f6a30f04e6b" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 37739 } }
I20260924 16:02:18.512897 31733 leader_election.cc:302] T 00000000000000000000000000000000 P 42171ced686844f4a7fd3f6a30f04e6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42171ced686844f4a7fd3f6a30f04e6b; no voters:
I20260924 16:02:18.513253 31733 leader_election.cc:288] T 00000000000000000000000000000000 P 42171ced686844f4a7fd3f6a30f04e6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:02:18.513322 31738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42171ced686844f4a7fd3f6a30f04e6b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:18.513556 31738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42171ced686844f4a7fd3f6a30f04e6b [term 1 LEADER]: Becoming Leader. State: Replica: 42171ced686844f4a7fd3f6a30f04e6b, State: Running, Role: LEADER
I20260924 16:02:18.513903 31738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42171ced686844f4a7fd3f6a30f04e6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42171ced686844f4a7fd3f6a30f04e6b" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 37739 } }
I20260924 16:02:18.514607 31733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42171ced686844f4a7fd3f6a30f04e6b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:02:18.515606 31740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42171ced686844f4a7fd3f6a30f04e6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42171ced686844f4a7fd3f6a30f04e6b. Latest consensus state: current_term: 1 leader_uuid: "42171ced686844f4a7fd3f6a30f04e6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42171ced686844f4a7fd3f6a30f04e6b" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 37739 } } }
I20260924 16:02:18.515805 31740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42171ced686844f4a7fd3f6a30f04e6b [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:18.516247 31747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:02:18.517031 31739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42171ced686844f4a7fd3f6a30f04e6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42171ced686844f4a7fd3f6a30f04e6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42171ced686844f4a7fd3f6a30f04e6b" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 37739 } } }
I20260924 16:02:18.517184 31739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42171ced686844f4a7fd3f6a30f04e6b [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:18.517867 31736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:18.504013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44034 (local address 127.25.109.62:37739)
0924 16:02:18.504494 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:18.504511 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:18.504571 (+ 60us) server_negotiation.cc:407] Connection header received
0924 16:02:18.505229 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:18.505246 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:18.505622 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:18.505947 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:18.506786 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:18.507797 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:18.508741 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:18.509214 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:18.509387 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:18.512149 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:18.512160 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:18.512173 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:18.512200 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:18.514270 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:18.514734 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:18.514744 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:18.514749 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:18.514863 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:18.515196 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:18.515199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:18.515201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:18.515729 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:18.515927 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:18.516701 (+ 774us) server_negotiation.cc:299] Negotiation successful
0924 16:02:18.517033 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":134,"threads_started":1}
I20260924 16:02:18.519987 31747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:02:18.525478 31747 catalog_manager.cc:1347] Generated new cluster ID: 2449641f2c3b4682be413201d6646b21
I20260924 16:02:18.525604 31747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:02:18.535722 31747 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:02:18.536718 31747 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:02:18.549252 31747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42171ced686844f4a7fd3f6a30f04e6b: Generated new TSK 0
I20260924 16:02:18.549953 31747 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
I20260924 16:02:18.580279 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:0
--local_ip_for_outbound_sockets=127.25.109.1
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:37739
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:18.692440 31759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:18.692708 31759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:18.692750 31759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:18.696794 31759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:18.696843 31759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:18.696900 31759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:02:18.702018 31759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:37739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:18.703038 31759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:18.704048 31759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:18.713786 31764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:18.713778 31765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:18.713891 31759 server_base.cc:1034] running on GCE node
W20260924 16:02:18.713778 31767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:18.714572 31759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:18.715380 31759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:18.716571 31759 hybrid_clock.cc:648] HybridClock initialized: now 1790265738716549 us; error 33 us; skew 500 ppm
Sep 24 16:02:18 dist-test-slave-n106 krb5kdc[31651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265738, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:18.719434 31759 init.cc:377] Logged in from keytab as kudu/127.25.109.1@KRBTEST.COM (short username kudu)
I20260924 16:02:18.720299 31759 webserver.cc:460] Webserver started at http://127.25.109.1:34435/ using document root <none> and password file <none>
I20260924 16:02:18.720950 31759 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:18.721017 31759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:18.721189 31759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:18.723269 31759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-0/data/instance:
uuid: "8857087ef0594cd5b2ce3c194e85a82c"
format_stamp: "Formatted at 2026-09-24 16:02:18 on dist-test-slave-n106"
server_key: "ddad21ecdaade2fad94009ca5bccdb89"
server_key_iv: "3ba5949c9f7d7f5dde283d2e4c15b5e9"
server_key_version: "encryptionkey@0"
I20260924 16:02:18.723821 31759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance:
uuid: "8857087ef0594cd5b2ce3c194e85a82c"
format_stamp: "Formatted at 2026-09-24 16:02:18 on dist-test-slave-n106"
server_key: "ddad21ecdaade2fad94009ca5bccdb89"
server_key_iv: "3ba5949c9f7d7f5dde283d2e4c15b5e9"
server_key_version: "encryptionkey@0"
I20260924 16:02:18.727639 31759 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 16:02:18.730146 31774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:18.731405 31759 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260924 16:02:18.731568 31759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "8857087ef0594cd5b2ce3c194e85a82c"
format_stamp: "Formatted at 2026-09-24 16:02:18 on dist-test-slave-n106"
server_key: "ddad21ecdaade2fad94009ca5bccdb89"
server_key_iv: "3ba5949c9f7d7f5dde283d2e4c15b5e9"
server_key_version: "encryptionkey@0"
I20260924 16:02:18.731670 31759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:18.749444 31759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:18.752566 31759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:18.752767 31759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:18.753408 31759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:18.754281 31759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:18.754328 31759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:18.754398 31759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:18.754431 31759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:18.763983 31759 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:42991
I20260924 16:02:18.764003 31887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:42991 every 8 connection(s)
I20260924 16:02:18.764990 31759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:02:18.766724 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 31759
I20260924 16:02:18.766870 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance
I20260924 16:02:18.767231 26036 external_mini_cluster.cc:1428] Setting key f7870bc6f087c8d0f36a23e071e6f1a3
Sep 24 16:02:18 dist-test-slave-n106 krb5kdc[31651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265738, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.2@KRBTEST.COM" created.
I20260924 16:02:18.781944 31736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:18.767004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:41217 (local address 127.25.109.62:37739)
0924 16:02:18.767182 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:18.767188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:18.768424 (+ 1236us) server_negotiation.cc:407] Connection header received
0924 16:02:18.770516 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:18.770519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:18.770566 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:18.770639 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:18.771968 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:18.772584 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:18.774550 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:18.774801 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:18.774889 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:18.777942 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:18.777946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:18.777948 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:18.777973 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:18.779426 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:18.780024 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:18.780028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:18.780029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:18.780148 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:18.780583 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:18.780587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:18.780589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:18.780846 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:18.780966 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:18.781521 (+ 555us) server_negotiation.cc:299] Negotiation successful
0924 16:02:18.781674 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 16:02:18.782867 31890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:18.767521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:37739 (local address 127.25.109.1:41217)
0924 16:02:18.768215 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:18.768299 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:18.770228 (+ 1929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:18.770881 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:18.770895 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:18.771331 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:18.771835 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:18.771849 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:18.772717 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:18.772724 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:18.774398 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:18.774415 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:18.774983 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:18.774990 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:18.775257 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:18.775956 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:18.775980 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:18.777748 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:18.779588 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:18.779594 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:18.779614 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:18.779903 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:18.780251 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:18.780254 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:18.780256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:18.780468 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:18.780986 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:18.780992 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:18.781255 (+ 263us) client_negotiation.cc:769] Sending connection context
0924 16:02:18.781420 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 16:02:18.781727 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":145,"threads_started":1}
I20260924 16:02:18.784297 31888 heartbeater.cc:347] Connected to a master server at 127.25.109.62:37739
I20260924 16:02:18.784610 31888 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:18.785153 31888 heartbeater.cc:511] Master 127.25.109.62:37739 requested a full tablet report, sending...
I20260924 16:02:18.786940 31697 ts_manager.cc:194] Registered new tserver with Master: 8857087ef0594cd5b2ce3c194e85a82c (127.25.109.1:42991)
I20260924 16:02:18.788651 31697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.1@KRBTEST.COM'} at 127.25.109.1:41217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
I20260924 16:02:18.797634 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:0
--local_ip_for_outbound_sockets=127.25.109.2
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:37739
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:18.906992 31893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:18.907231 31893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:18.907272 31893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:18.911338 31893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:18.911384 31893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:18.911475 31893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:02:18.916834 31893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:37739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:18.917809 31893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:18.918798 31893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:18.928282 31899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:18.928344 31893 server_base.cc:1034] running on GCE node
W20260924 16:02:18.928510 31901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:18.928295 31898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:18.929029 31893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:18.929651 31893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:18.930835 31893 hybrid_clock.cc:648] HybridClock initialized: now 1790265738930828 us; error 32 us; skew 500 ppm
Sep 24 16:02:18 dist-test-slave-n106 krb5kdc[31651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265738, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:18.933554 31893 init.cc:377] Logged in from keytab as kudu/127.25.109.2@KRBTEST.COM (short username kudu)
I20260924 16:02:18.934376 31893 webserver.cc:460] Webserver started at http://127.25.109.2:40903/ using document root <none> and password file <none>
I20260924 16:02:18.934983 31893 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:18.935073 31893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:18.935305 31893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:18.937428 31893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-1/data/instance:
uuid: "622e51abe137496896acc962234cf0c9"
format_stamp: "Formatted at 2026-09-24 16:02:18 on dist-test-slave-n106"
server_key: "84263ac67f94e64fec12ee11ea3ffccb"
server_key_iv: "9c9d1c12bf9acd3444be377949d0d7f9"
server_key_version: "encryptionkey@0"
I20260924 16:02:18.937994 31893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance:
uuid: "622e51abe137496896acc962234cf0c9"
format_stamp: "Formatted at 2026-09-24 16:02:18 on dist-test-slave-n106"
server_key: "84263ac67f94e64fec12ee11ea3ffccb"
server_key_iv: "9c9d1c12bf9acd3444be377949d0d7f9"
server_key_version: "encryptionkey@0"
I20260924 16:02:18.941632 31893 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:02:18.943958 31908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:18.945178 31893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:02:18.945313 31893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "622e51abe137496896acc962234cf0c9"
format_stamp: "Formatted at 2026-09-24 16:02:18 on dist-test-slave-n106"
server_key: "84263ac67f94e64fec12ee11ea3ffccb"
server_key_iv: "9c9d1c12bf9acd3444be377949d0d7f9"
server_key_version: "encryptionkey@0"
I20260924 16:02:18.945425 31893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:18.958016 31893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:18.960734 31893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:18.960933 31893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:18.961488 31893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:18.962306 31893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:18.962378 31893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:18.962447 31893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:18.962495 31893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:18.971566 31893 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:37325
I20260924 16:02:18.971602 32021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:37325 every 8 connection(s)
I20260924 16:02:18.972445 31893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:02:18.973515 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 31893
I20260924 16:02:18.973649 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance
I20260924 16:02:18.973980 26036 external_mini_cluster.cc:1428] Setting key ae0c10ec55becc65c638c43bc015d6e1
Sep 24 16:02:18 dist-test-slave-n106 krb5kdc[31651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265738, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.3@KRBTEST.COM" created.
I20260924 16:02:18.987649 31736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:18.974374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:38551 (local address 127.25.109.62:37739)
0924 16:02:18.974491 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:18.974498 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:18.975469 (+ 971us) server_negotiation.cc:407] Connection header received
0924 16:02:18.976892 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:18.976897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:18.976964 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:18.977071 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:18.978634 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:18.979522 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:18.980791 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:18.981256 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:18.981398 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:18.983796 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:18.983801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:18.983804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:18.983830 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:18.985353 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:18.985866 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:18.985869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:18.985870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:18.985948 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:18.986245 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:18.986247 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:18.986248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:18.986468 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:18.986592 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:18.987237 (+ 645us) server_negotiation.cc:299] Negotiation successful
0924 16:02:18.987378 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 16:02:18.988333 32024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:18.974834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:37739 (local address 127.25.109.2:38551)
0924 16:02:18.975320 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:18.975371 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:18.976697 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:18.977296 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:18.977307 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:18.977809 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:18.978479 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:18.978496 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:18.979675 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:18.979680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:18.980667 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:18.980678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:18.981409 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:18.981412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:18.981546 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:18.982143 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:18.982159 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:18.983631 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:18.985460 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:18.985464 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:18.985476 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:18.985741 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:18.986032 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:18.986034 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:18.986035 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:18.986165 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:18.986607 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:18.986616 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:18.986928 (+ 312us) client_negotiation.cc:769] Sending connection context
0924 16:02:18.987150 (+ 222us) client_negotiation.cc:240] Negotiation successful
0924 16:02:18.987434 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":130,"threads_started":1}
I20260924 16:02:18.989723 32022 heartbeater.cc:347] Connected to a master server at 127.25.109.62:37739
I20260924 16:02:18.989987 32022 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:18.990630 32022 heartbeater.cc:511] Master 127.25.109.62:37739 requested a full tablet report, sending...
I20260924 16:02:18.991829 31697 ts_manager.cc:194] Registered new tserver with Master: 622e51abe137496896acc962234cf0c9 (127.25.109.2:37325)
I20260924 16:02:18.992480 31697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.2@KRBTEST.COM'} at 127.25.109.2:38551
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
I20260924 16:02:19.002709 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:0
--local_ip_for_outbound_sockets=127.25.109.3
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:37739
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:19.112701 32027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:19.112949 32027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:19.112990 32027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:19.117138 32027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:19.117193 32027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:19.117290 32027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:02:19.122524 32027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:37739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:19.123605 32027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:19.124850 32027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:19.134547 32032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:19.134639 32035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:19.134653 32033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:19.134859 32027 server_base.cc:1034] running on GCE node
I20260924 16:02:19.135260 32027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:19.135960 32027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:19.137156 32027 hybrid_clock.cc:648] HybridClock initialized: now 1790265739137124 us; error 45 us; skew 500 ppm
Sep 24 16:02:19 dist-test-slave-n106 krb5kdc[31651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265739, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:19.140050 32027 init.cc:377] Logged in from keytab as kudu/127.25.109.3@KRBTEST.COM (short username kudu)
I20260924 16:02:19.140956 32027 webserver.cc:460] Webserver started at http://127.25.109.3:41471/ using document root <none> and password file <none>
I20260924 16:02:19.141533 32027 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:19.141623 32027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:19.141853 32027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:19.143872 32027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-2/data/instance:
uuid: "a2e82b783ea84d3bbe1e5faacad092f3"
format_stamp: "Formatted at 2026-09-24 16:02:19 on dist-test-slave-n106"
server_key: "13b3cf65d99583a934835bf5bfac28ee"
server_key_iv: "de5ea9770940ad676f9385d41adc8359"
server_key_version: "encryptionkey@0"
I20260924 16:02:19.144498 32027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance:
uuid: "a2e82b783ea84d3bbe1e5faacad092f3"
format_stamp: "Formatted at 2026-09-24 16:02:19 on dist-test-slave-n106"
server_key: "13b3cf65d99583a934835bf5bfac28ee"
server_key_iv: "de5ea9770940ad676f9385d41adc8359"
server_key_version: "encryptionkey@0"
I20260924 16:02:19.148128 32027 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:02:19.150665 32042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:19.151844 32027 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:02:19.151983 32027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "a2e82b783ea84d3bbe1e5faacad092f3"
format_stamp: "Formatted at 2026-09-24 16:02:19 on dist-test-slave-n106"
server_key: "13b3cf65d99583a934835bf5bfac28ee"
server_key_iv: "de5ea9770940ad676f9385d41adc8359"
server_key_version: "encryptionkey@0"
I20260924 16:02:19.152092 32027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:19.179373 32027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:19.182471 32027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:19.182682 32027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:19.183282 32027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:19.184157 32027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:19.184230 32027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:19.184303 32027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:19.184347 32027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:19.193483 32027 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:41573
I20260924 16:02:19.193516 32155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:41573 every 8 connection(s)
I20260924 16:02:19.194402 32027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:02:19.198763 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 32027
I20260924 16:02:19.198885 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance
I20260924 16:02:19.199205 26036 external_mini_cluster.cc:1428] Setting key 3999e54ff3bfa9831ea971df958602c4
Sep 24 16:02:19 dist-test-slave-n106 krb5kdc[31651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265739, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:02:19.208340 31736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:19.196320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:58387 (local address 127.25.109.62:37739)
0924 16:02:19.196520 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:19.196524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:19.197307 (+ 783us) server_negotiation.cc:407] Connection header received
0924 16:02:19.198696 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:19.198700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:19.198750 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:19.198867 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:19.200292 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:19.200921 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:19.202221 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:19.202465 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:19.202574 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:19.205044 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:19.205047 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:19.205049 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:19.205074 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:19.206362 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:19.206828 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:19.206830 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:19.206832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:19.206911 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:19.207382 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:19.207385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:19.207387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:19.207578 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:19.207660 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:19.208114 (+ 454us) server_negotiation.cc:299] Negotiation successful
0924 16:02:19.208219 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 16:02:19.209163 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:19.196714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:37739 (local address 127.25.109.3:58387)
0924 16:02:19.197152 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:19.197195 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:19.198395 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:19.199032 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:19.199043 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:19.199545 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:19.200124 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:19.200142 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:19.201085 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:19.201089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:19.202086 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:19.202096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:19.202624 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:19.202629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:19.202805 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:19.203430 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:19.203448 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:19.204887 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:19.206483 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:19.206488 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:19.206499 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:19.206742 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:19.207012 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:19.207015 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:19.207017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:19.207233 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:19.207678 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:19.207683 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:19.207901 (+ 218us) client_negotiation.cc:769] Sending connection context
0924 16:02:19.208060 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 16:02:19.208298 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":98,"threads_started":1}
I20260924 16:02:19.210274 32156 heartbeater.cc:347] Connected to a master server at 127.25.109.62:37739
I20260924 16:02:19.210549 32156 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:19.211041 32156 heartbeater.cc:511] Master 127.25.109.62:37739 requested a full tablet report, sending...
I20260924 16:02:19.212136 31697 ts_manager.cc:194] Registered new tserver with Master: a2e82b783ea84d3bbe1e5faacad092f3 (127.25.109.3:41573)
I20260924 16:02:19.212881 31697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.3@KRBTEST.COM'} at 127.25.109.3:58387
I20260924 16:02:19.213188 26036 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/test-admin.keytab.
Sep 24 16:02:19 dist-test-slave-n106 krb5kdc[31651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:19.228955 26036 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 16:02:19.231662 32163 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:19.230253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40016 (local address 127.25.109.1:42991)
0924 16:02:19.230538 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:19.230551 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:19.230574 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:02:19.230663 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:19.230667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:19.230845 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:19.230961 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:19.231458 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40016: BlockingRecv error: recv got EOF from 127.0.0.1:40016 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":82,"threads_started":1}
W20260924 16:02:19.231694 26036 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/krb5cc' of type FILE
Sep 24 16:02:19 dist-test-slave-n106 krb5kdc[31651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:19.232734 26036 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 16:02:19 dist-test-slave-n106 krb5kdc[31651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:19.233695 26036 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 16:02:19 dist-test-slave-n106 krb5kdc[31651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.1@KRBTEST.COM
I20260924 16:02:19.243709 32163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:19.234504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40032 (local address 127.25.109.1:42991)
0924 16:02:19.234636 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:19.234642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:19.234742 (+ 100us) server_negotiation.cc:407] Connection header received
0924 16:02:19.234814 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:19.234816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:19.234853 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:19.234918 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:19.235469 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:19.236328 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:19.237331 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:19.237781 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:19.237867 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:19.240305 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:19.240314 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:19.240327 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:19.240350 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:19.241924 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:19.242350 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:19.242356 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:19.242357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:19.242433 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:19.242678 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:19.242681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:19.242682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:19.242909 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:19.243055 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:19.243357 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 16:02:19.243536 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 16:02:19.247586 32163 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:19.246740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40040 (local address 127.25.109.1:42991)
0924 16:02:19.246876 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:19.246879 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:19.246907 (+ 28us) server_negotiation.cc:407] Connection header received
0924 16:02:19.247022 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:19.247024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:19.247059 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:19.247124 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:19.247489 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40040: BlockingRecv error: recv got EOF from 127.0.0.1:40040 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 16:02:19.247723 26036 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/krb5cc' of type FILE
Sep 24 16:02:19 dist-test-slave-n106 krb5kdc[31651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:19.248894 26036 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 16:02:19 dist-test-slave-n106 krb5kdc[31651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:19.249938 26036 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 16:02:19 dist-test-slave-n106 krb5kdc[31651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.1@KRBTEST.COM
I20260924 16:02:19.258335 32163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:19.250778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40050 (local address 127.25.109.1:42991)
0924 16:02:19.250920 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:19.250924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:19.251044 (+ 120us) server_negotiation.cc:407] Connection header received
0924 16:02:19.251190 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:19.251194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:19.251249 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:19.251329 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:19.251848 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:19.252373 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:19.253215 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:19.253441 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:19.253523 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:19.255599 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:19.255603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:19.255604 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:19.255625 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:19.256968 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:19.257345 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:19.257348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:19.257349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:19.257417 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:19.257638 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:19.257640 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:19.257641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:19.257786 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:19.257876 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:19.258076 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 16:02:19.258159 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 16:02:19.261090 32163 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:19.260088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40052 (local address 127.25.109.1:42991)
0924 16:02:19.260217 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:19.260221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:19.260339 (+ 118us) server_negotiation.cc:407] Connection header received
0924 16:02:19.260495 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:19.260498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:19.260544 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:19.260626 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:19.260992 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40052: BlockingRecv error: recv got EOF from 127.0.0.1:40052 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 16:02:19.261216 26036 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265708917108-26036-0/krb5kdc/krb5cc' of type FILE
Sep 24 16:02:19 dist-test-slave-n106 krb5kdc[31651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:19.262151 26036 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 16:02:19 dist-test-slave-n106 krb5kdc[31651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:19.262961 26036 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 16:02:19 dist-test-slave-n106 krb5kdc[31651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.1@KRBTEST.COM
I20260924 16:02:19.270682 32163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:19.263715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40066 (local address 127.25.109.1:42991)
0924 16:02:19.263831 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:19.263834 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:19.263879 (+ 45us) server_negotiation.cc:407] Connection header received
0924 16:02:19.263963 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:19.263965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:19.263996 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:19.264056 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:19.264525 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:19.264991 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:19.265706 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:19.265901 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:19.265972 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:19.267893 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:19.267896 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:19.267898 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:19.267918 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:19.269263 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:19.269691 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:19.269694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:19.269695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:19.269768 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:19.269989 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:19.269992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:19.269993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:19.270132 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:19.270218 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:19.270416 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 16:02:19.270531 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 16:02:19.271633 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 31759
I20260924 16:02:19.277772 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 31893
I20260924 16:02:19.283421 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 32027
I20260924 16:02:19.289033 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 31664
2026-09-24T16:02:19Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2151 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:02:19 dist-test-slave-n106 krb5kdc[32176](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:02:19 dist-test-slave-n106 krb5kdc[32176](info): set up 2 sockets
Sep 24 16:02:19 dist-test-slave-n106 krb5kdc[32176](info): commencing operation
krb5kdc: starting...
W20260924 16:02:20.329711 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:02:20 dist-test-slave-n106 krb5kdc[32176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:02:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:02:20Z Disabled control of system clock
WARNING: no policy specified for oryx/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.62.keytab.
Entry for principal oryx/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.62.keytab.
I20260924 16:02:20.450704 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:33903
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.62.keytab
--principal=oryx/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:33903
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:20.560839 32189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:20.561167 32189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:20.561242 32189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:20.565258 32189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:02:20.565356 32189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:20.565394 32189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:20.565429 32189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:02:20.565465 32189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:02:20.570858 32189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:33903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.62.keytab
--principal=oryx/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:33903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:20.572000 32189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:20.573096 32189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:20.581568 32195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:20.581734 32189 server_base.cc:1034] running on GCE node
W20260924 16:02:20.581616 32197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:20.581913 32194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:20.582512 32189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:20.583688 32189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:20.584883 32189 hybrid_clock.cc:648] HybridClock initialized: now 1790265740584863 us; error 30 us; skew 500 ppm
Sep 24 16:02:20 dist-test-slave-n106 krb5kdc[32176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265740, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:20.587653 32189 init.cc:377] Logged in from keytab as oryx/127.25.109.62@KRBTEST.COM (short username oryx)
I20260924 16:02:20.588665 32189 webserver.cc:460] Webserver started at http://127.25.109.62:33247/ using document root <none> and password file <none>
I20260924 16:02:20.589250 32189 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:20.589342 32189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:20.589582 32189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:20.591643 32189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "bda7fa2c9ca5483ea36450964edfe158"
format_stamp: "Formatted at 2026-09-24 16:02:20 on dist-test-slave-n106"
server_key: "8e0bd48caa702f75ff912e8b3c4fe70b"
server_key_iv: "463792406e050b63f3f37fc3c3e2461b"
server_key_version: "encryptionkey@0"
I20260924 16:02:20.592263 32189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "bda7fa2c9ca5483ea36450964edfe158"
format_stamp: "Formatted at 2026-09-24 16:02:20 on dist-test-slave-n106"
server_key: "8e0bd48caa702f75ff912e8b3c4fe70b"
server_key_iv: "463792406e050b63f3f37fc3c3e2461b"
server_key_version: "encryptionkey@0"
I20260924 16:02:20.596104 32189 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260924 16:02:20.598732 32204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:20.600015 32189 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 16:02:20.600185 32189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "bda7fa2c9ca5483ea36450964edfe158"
format_stamp: "Formatted at 2026-09-24 16:02:20 on dist-test-slave-n106"
server_key: "8e0bd48caa702f75ff912e8b3c4fe70b"
server_key_iv: "463792406e050b63f3f37fc3c3e2461b"
server_key_version: "encryptionkey@0"
I20260924 16:02:20.600312 32189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:20.618628 32189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:20.621670 32189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:20.621879 32189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:20.629256 32189 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:33903
I20260924 16:02:20.629568 32256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:33903 every 8 connection(s)
I20260924 16:02:20.630267 32189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:02:20.633100 32257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:20.636570 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 32189
I20260924 16:02:20.636713 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:02:20.637030 26036 external_mini_cluster.cc:1428] Setting key a421fea6805a055fd5bb04a11665cd21
I20260924 16:02:20.639271 32257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158: Bootstrap starting.
I20260924 16:02:20.641554 32257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:20.642302 32257 log.cc:824] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:20.644209 32257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158: No bootstrap required, opened a new log
Sep 24 16:02:20 dist-test-slave-n106 krb5kdc[32176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM
I20260924 16:02:20.648499 32257 raft_consensus.cc:348] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } }
I20260924 16:02:20.648716 32257 raft_consensus.cc:374] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:20.648777 32257 raft_consensus.cc:729] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda7fa2c9ca5483ea36450964edfe158, State: Initialized, Role: FOLLOWER
I20260924 16:02:20.649248 32257 consensus_queue.cc:260] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } }
I20260924 16:02:20.649384 32257 raft_consensus.cc:388] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:02:20.649467 32257 raft_consensus.cc:482] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:02:20.649572 32257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:20.650465 32257 raft_consensus.cc:504] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } }
I20260924 16:02:20.650873 32257 leader_election.cc:302] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bda7fa2c9ca5483ea36450964edfe158; no voters:
I20260924 16:02:20.651250 32257 leader_election.cc:288] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:02:20.651322 32262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:20.651536 32262 raft_consensus.cc:686] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [term 1 LEADER]: Becoming Leader. State: Replica: bda7fa2c9ca5483ea36450964edfe158, State: Running, Role: LEADER
I20260924 16:02:20.651837 32262 consensus_queue.cc:237] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } }
I20260924 16:02:20.651885 32260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:20.638524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41074 (local address 127.25.109.62:33903)
0924 16:02:20.638986 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:20.639001 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:20.639038 (+ 37us) server_negotiation.cc:407] Connection header received
0924 16:02:20.639728 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:20.639749 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:20.640095 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:20.640456 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:20.641318 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:20.642395 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:20.643491 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:20.643969 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:20.644150 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:20.646835 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:20.646841 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:20.646853 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:20.646880 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:20.648743 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:20.649217 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:20.649225 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:20.649236 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:20.649398 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:20.649762 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:20.649766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:20.649768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:20.650169 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:20.650380 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:20.650624 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 16:02:20.650914 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":134,"threads_started":1}
I20260924 16:02:20.652616 32257 sys_catalog.cc:564] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:02:20.653473 32263 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bda7fa2c9ca5483ea36450964edfe158. Latest consensus state: current_term: 1 leader_uuid: "bda7fa2c9ca5483ea36450964edfe158" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } }
I20260924 16:02:20.653601 32263 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:20.653963 32264 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bda7fa2c9ca5483ea36450964edfe158" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } }
I20260924 16:02:20.654060 32264 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:20.654040 32266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:02:20.657510 32266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:02:20.663134 32266 catalog_manager.cc:1347] Generated new cluster ID: 3cb794533a7b41a6955bf42add1fa079
I20260924 16:02:20.663201 32266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:02:20.688735 32266 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:02:20.690060 32266 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:02:20.705551 32266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158: Generated new TSK 0
I20260924 16:02:20.706432 32266 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.25.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.1.keytab.
Entry for principal oryx/127.25.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.1.keytab.
I20260924 16:02:20.744204 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:0
--local_ip_for_outbound_sockets=127.25.109.1
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:33903
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.1.keytab
--principal=oryx/127.25.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:20.860108 32284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:20.860484 32284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:20.860605 32284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:20.864755 32284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:20.864841 32284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:20.864935 32284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:02:20.870378 32284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.1.keytab
--principal=oryx/127.25.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:33903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:20.871529 32284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:20.872654 32284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:20.882488 32290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:20.882488 32289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:20.882700 32284 server_base.cc:1034] running on GCE node
W20260924 16:02:20.882557 32292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:20.883190 32284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:20.883919 32284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:20.885104 32284 hybrid_clock.cc:648] HybridClock initialized: now 1790265740885066 us; error 54 us; skew 500 ppm
Sep 24 16:02:20 dist-test-slave-n106 krb5kdc[32176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265740, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:20.887904 32284 init.cc:377] Logged in from keytab as oryx/127.25.109.1@KRBTEST.COM (short username oryx)
I20260924 16:02:20.888885 32284 webserver.cc:460] Webserver started at http://127.25.109.1:43059/ using document root <none> and password file <none>
I20260924 16:02:20.889487 32284 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:20.889587 32284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:20.889832 32284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:20.891981 32284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/data/instance:
uuid: "c3fa4c2ab3d94a049605e22b6492d8ab"
format_stamp: "Formatted at 2026-09-24 16:02:20 on dist-test-slave-n106"
server_key: "48254c4cc748111afcf2d4f4deafce3d"
server_key_iv: "37d3d00ebb8afb5c3cd5933745f66246"
server_key_version: "encryptionkey@0"
I20260924 16:02:20.892612 32284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance:
uuid: "c3fa4c2ab3d94a049605e22b6492d8ab"
format_stamp: "Formatted at 2026-09-24 16:02:20 on dist-test-slave-n106"
server_key: "48254c4cc748111afcf2d4f4deafce3d"
server_key_iv: "37d3d00ebb8afb5c3cd5933745f66246"
server_key_version: "encryptionkey@0"
I20260924 16:02:20.896617 32284 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:02:20.899133 32299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:20.900465 32284 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:02:20.900626 32284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "c3fa4c2ab3d94a049605e22b6492d8ab"
format_stamp: "Formatted at 2026-09-24 16:02:20 on dist-test-slave-n106"
server_key: "48254c4cc748111afcf2d4f4deafce3d"
server_key_iv: "37d3d00ebb8afb5c3cd5933745f66246"
server_key_version: "encryptionkey@0"
I20260924 16:02:20.900756 32284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:20.916692 32284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:20.919770 32284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:20.919979 32284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:20.920739 32284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:20.921595 32284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:20.921665 32284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:20.921737 32284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:20.921782 32284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:20.930451 32284 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:46785
I20260924 16:02:20.930475 32412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:46785 every 8 connection(s)
I20260924 16:02:20.931406 32284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:02:20.941079 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 32284
I20260924 16:02:20.941187 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance
Sep 24 16:02:20 dist-test-slave-n106 krb5kdc[32176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265740, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM
I20260924 16:02:20.941557 26036 external_mini_cluster.cc:1428] Setting key 620f6666ed623b30d6d8fedef485e417
I20260924 16:02:20.945747 32260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:20.933332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:47191 (local address 127.25.109.62:33903)
0924 16:02:20.933482 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:20.933486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:20.934344 (+ 858us) server_negotiation.cc:407] Connection header received
0924 16:02:20.935743 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:20.935747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:20.935809 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:20.935914 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:20.937172 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:20.937721 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:20.938861 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:20.939112 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:20.939225 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:20.941928 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:20.941933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:20.941936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:20.941969 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:20.943526 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:20.944056 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:20.944063 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:20.944068 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:20.944174 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:20.944539 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:20.944545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:20.944550 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:20.944817 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:20.944923 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:20.945445 (+ 522us) server_negotiation.cc:299] Negotiation successful
0924 16:02:20.945578 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:02:20.946569 32415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:20.933733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.25.109.1:47191)
0924 16:02:20.934195 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:20.934238 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:20.935523 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:20.936106 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:20.936112 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:20.936528 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:20.937022 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:20.937038 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:20.937855 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:20.937858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:20.938734 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:20.938743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:20.939271 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:20.939274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:20.939449 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:20.940112 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:20.940128 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:20.941756 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:20.943648 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:20.943653 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:20.943667 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:20.943946 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:20.944270 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:20.944273 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:20.944274 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:20.944454 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:20.944922 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:20.944946 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:20.945192 (+ 246us) client_negotiation.cc:769] Sending connection context
0924 16:02:20.945384 (+ 192us) client_negotiation.cc:240] Negotiation successful
0924 16:02:20.945734 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":118,"threads_started":1}
I20260924 16:02:20.947830 32413 heartbeater.cc:347] Connected to a master server at 127.25.109.62:33903
I20260924 16:02:20.948087 32413 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:20.948648 32413 heartbeater.cc:511] Master 127.25.109.62:33903 requested a full tablet report, sending...
I20260924 16:02:20.950376 32221 ts_manager.cc:194] Registered new tserver with Master: c3fa4c2ab3d94a049605e22b6492d8ab (127.25.109.1:46785)
WARNING: no policy specified for oryx/127.25.109.2@KRBTEST.COM; defaulting to no policy
I20260924 16:02:20.952112 32221 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.109.1@KRBTEST.COM'} at 127.25.109.1:47191
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.2.keytab.
Entry for principal oryx/127.25.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.2.keytab.
I20260924 16:02:20.969719 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:0
--local_ip_for_outbound_sockets=127.25.109.2
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:33903
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.2.keytab
--principal=oryx/127.25.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:21.080860 32418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:21.081167 32418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:21.081213 32418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:21.085347 32418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:21.085405 32418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:21.085465 32418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:02:21.090965 32418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.2.keytab
--principal=oryx/127.25.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:33903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:21.092114 32418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:21.093299 32418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:21.103066 32423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:21.103081 32424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:21.103087 32426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:21.103652 32418 server_base.cc:1034] running on GCE node
I20260924 16:02:21.104076 32418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:21.104846 32418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:21.106045 32418 hybrid_clock.cc:648] HybridClock initialized: now 1790265741106024 us; error 33 us; skew 500 ppm
Sep 24 16:02:21 dist-test-slave-n106 krb5kdc[32176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265741, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:21.108949 32418 init.cc:377] Logged in from keytab as oryx/127.25.109.2@KRBTEST.COM (short username oryx)
I20260924 16:02:21.109894 32418 webserver.cc:460] Webserver started at http://127.25.109.2:33049/ using document root <none> and password file <none>
I20260924 16:02:21.110486 32418 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:21.110585 32418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:21.110808 32418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:21.112967 32418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/data/instance:
uuid: "f758e6c10f3448cdb5d1e3ba31d30925"
format_stamp: "Formatted at 2026-09-24 16:02:21 on dist-test-slave-n106"
server_key: "cc027d32ee6955225589ca1c58e845d9"
server_key_iv: "6f1d3fd423142ffd5acea9fd58bf281a"
server_key_version: "encryptionkey@0"
I20260924 16:02:21.113561 32418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance:
uuid: "f758e6c10f3448cdb5d1e3ba31d30925"
format_stamp: "Formatted at 2026-09-24 16:02:21 on dist-test-slave-n106"
server_key: "cc027d32ee6955225589ca1c58e845d9"
server_key_iv: "6f1d3fd423142ffd5acea9fd58bf281a"
server_key_version: "encryptionkey@0"
I20260924 16:02:21.117424 32418 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:02:21.119951 32433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:21.121312 32418 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:02:21.121603 32418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "f758e6c10f3448cdb5d1e3ba31d30925"
format_stamp: "Formatted at 2026-09-24 16:02:21 on dist-test-slave-n106"
server_key: "cc027d32ee6955225589ca1c58e845d9"
server_key_iv: "6f1d3fd423142ffd5acea9fd58bf281a"
server_key_version: "encryptionkey@0"
I20260924 16:02:21.121778 32418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:21.143826 32418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:21.147089 32418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:21.147306 32418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:21.147956 32418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:21.148932 32418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:21.149008 32418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:21.149081 32418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:21.149130 32418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:21.158866 32418 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:38957
I20260924 16:02:21.158883 32546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:38957 every 8 connection(s)
I20260924 16:02:21.159827 32418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:02:21.166002 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 32418
I20260924 16:02:21.166110 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance
I20260924 16:02:21.166445 26036 external_mini_cluster.cc:1428] Setting key e6285718c4437f087fa3e03672c26ff3
Sep 24 16:02:21 dist-test-slave-n106 krb5kdc[32176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265741, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM
I20260924 16:02:21.174631 32260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:21.161741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:55731 (local address 127.25.109.62:33903)
0924 16:02:21.161908 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:21.161913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:21.162802 (+ 889us) server_negotiation.cc:407] Connection header received
0924 16:02:21.164191 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:21.164194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:21.164247 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:21.164329 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:21.165700 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.166304 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.167789 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.168106 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.168212 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:21.170834 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:21.170838 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:21.170840 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:21.170865 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:21.172369 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:21.172979 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:21.172989 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:21.172990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:21.173069 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:21.173465 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:21.173468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:21.173470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:21.173677 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:21.173787 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:21.174330 (+ 543us) server_negotiation.cc:299] Negotiation successful
0924 16:02:21.174463 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 16:02:21.175519 32549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:21.162191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.25.109.2:55731)
0924 16:02:21.162649 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:21.162692 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:21.163958 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:21.164681 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:21.164688 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:21.165050 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:21.165546 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:21.165560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.166431 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.166435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:21.167652 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:21.167666 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.168281 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.168284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:21.168512 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:21.169163 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:21.169179 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:21.170644 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:21.172526 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:21.172533 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:21.172550 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:21.172837 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:21.173176 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:21.173180 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:21.173182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:21.173348 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:21.173803 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:21.173811 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:21.174080 (+ 269us) client_negotiation.cc:769] Sending connection context
0924 16:02:21.174286 (+ 206us) client_negotiation.cc:240] Negotiation successful
0924 16:02:21.174646 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for oryx/127.25.109.3@KRBTEST.COM; defaulting to no policy
I20260924 16:02:21.176854 32547 heartbeater.cc:347] Connected to a master server at 127.25.109.62:33903
I20260924 16:02:21.177130 32547 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:21.177660 32547 heartbeater.cc:511] Master 127.25.109.62:33903 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.109.3@KRBTEST.COM" created.
I20260924 16:02:21.178884 32221 ts_manager.cc:194] Registered new tserver with Master: f758e6c10f3448cdb5d1e3ba31d30925 (127.25.109.2:38957)
I20260924 16:02:21.179793 32221 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.109.2@KRBTEST.COM'} at 127.25.109.2:55731
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.3.keytab.
Entry for principal oryx/127.25.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.3.keytab.
I20260924 16:02:21.195209 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:0
--local_ip_for_outbound_sockets=127.25.109.3
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:33903
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.3.keytab
--principal=oryx/127.25.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:21.307219 32552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:21.307502 32552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:21.307578 32552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:21.311528 32552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:21.311619 32552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:21.311717 32552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:02:21.317097 32552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.3.keytab
--principal=oryx/127.25.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:33903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:21.318213 32552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:21.319259 32552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:21.329093 32558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:21.329051 32557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:21.329061 32560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:21.329074 32552 server_base.cc:1034] running on GCE node
I20260924 16:02:21.329721 32552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:21.330377 32552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:21.331557 32552 hybrid_clock.cc:648] HybridClock initialized: now 1790265741331530 us; error 47 us; skew 500 ppm
Sep 24 16:02:21 dist-test-slave-n106 krb5kdc[32176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265741, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:21.334221 32552 init.cc:377] Logged in from keytab as oryx/127.25.109.3@KRBTEST.COM (short username oryx)
I20260924 16:02:21.335058 32552 webserver.cc:460] Webserver started at http://127.25.109.3:38853/ using document root <none> and password file <none>
I20260924 16:02:21.335562 32552 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:21.335628 32552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:21.335786 32552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:21.337810 32552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/data/instance:
uuid: "1c9c5af925794483afe9f26d1a0f4291"
format_stamp: "Formatted at 2026-09-24 16:02:21 on dist-test-slave-n106"
server_key: "bca5284f64488c93897c1eae1ce90559"
server_key_iv: "9aeb2ecd0612215437eb508a503ec12f"
server_key_version: "encryptionkey@0"
I20260924 16:02:21.338336 32552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance:
uuid: "1c9c5af925794483afe9f26d1a0f4291"
format_stamp: "Formatted at 2026-09-24 16:02:21 on dist-test-slave-n106"
server_key: "bca5284f64488c93897c1eae1ce90559"
server_key_iv: "9aeb2ecd0612215437eb508a503ec12f"
server_key_version: "encryptionkey@0"
I20260924 16:02:21.341956 32552 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260924 16:02:21.344363 32567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:21.345655 32552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 16:02:21.345752 32552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "1c9c5af925794483afe9f26d1a0f4291"
format_stamp: "Formatted at 2026-09-24 16:02:21 on dist-test-slave-n106"
server_key: "bca5284f64488c93897c1eae1ce90559"
server_key_iv: "9aeb2ecd0612215437eb508a503ec12f"
server_key_version: "encryptionkey@0"
I20260924 16:02:21.345887 32552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:21.370730 32552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:21.374080 32552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:21.374317 32552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:21.374972 32552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:21.375802 32552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:21.375857 32552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:21.375900 32552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:21.375921 32552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:21.385190 32552 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:41657
I20260924 16:02:21.385210 32680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:41657 every 8 connection(s)
I20260924 16:02:21.386094 32552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:02:21.391294 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 32552
I20260924 16:02:21.391433 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance
I20260924 16:02:21.391788 26036 external_mini_cluster.cc:1428] Setting key 968f02654e62a6b9a356348436c32f73
Sep 24 16:02:21 dist-test-slave-n106 krb5kdc[32176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265741, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM
I20260924 16:02:21.401252 32260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:21.387883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:36077 (local address 127.25.109.62:33903)
0924 16:02:21.388058 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:21.388063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:21.388955 (+ 892us) server_negotiation.cc:407] Connection header received
0924 16:02:21.390340 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:21.390343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:21.390395 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:21.390472 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:21.391989 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.392721 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.394439 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.394669 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.394757 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:21.397575 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:21.397578 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:21.397580 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:21.397607 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:21.399129 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:21.399626 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:21.399629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:21.399630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:21.399714 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:21.400056 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:21.400058 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:21.400060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:21.400247 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:21.400327 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:21.401008 (+ 681us) server_negotiation.cc:299] Negotiation successful
0924 16:02:21.401122 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 16:02:21.402156 32683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:21.388288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.25.109.3:36077)
0924 16:02:21.388799 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:21.388842 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:21.390135 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:21.390647 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:21.390654 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:21.391096 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:21.391817 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:21.391843 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.392857 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.392860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:21.394328 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:21.394339 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.394809 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.394812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:21.394957 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:21.395526 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:21.395543 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:21.397382 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:21.399243 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:21.399248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:21.399262 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:21.399533 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:21.399807 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:21.399810 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:21.399811 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:21.399965 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:21.400378 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:21.400384 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:21.400742 (+ 358us) client_negotiation.cc:769] Sending connection context
0924 16:02:21.400928 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 16:02:21.401227 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":179,"threads_started":1}
I20260924 16:02:21.403702 32681 heartbeater.cc:347] Connected to a master server at 127.25.109.62:33903
I20260924 16:02:21.404039 32681 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:21.404835 32681 heartbeater.cc:511] Master 127.25.109.62:33903 requested a full tablet report, sending...
I20260924 16:02:21.406068 32221 ts_manager.cc:194] Registered new tserver with Master: 1c9c5af925794483afe9f26d1a0f4291 (127.25.109.3:41657)
I20260924 16:02:21.406852 32221 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.109.3@KRBTEST.COM'} at 127.25.109.3:36077
I20260924 16:02:21.418678 26036 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.25.109.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.109.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.109.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.61.keytab.
Entry for principal oryx/127.25.109.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.61.keytab.
I20260924 16:02:21.447535 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.61:43277
--webserver_interface=127.25.109.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.61.keytab
--principal=oryx/127.25.109.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:33903,127.25.109.61:43277
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:21.565523 32686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:21.565797 32686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:21.565876 32686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:21.569971 32686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:02:21.570031 32686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:21.570051 32686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:21.570106 32686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:02:21.570138 32686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:02:21.575696 32686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:33903,127.25.109.61:43277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.61.keytab
--principal=oryx/127.25.109.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.61:43277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.109.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
W20260924 16:02:21.575938 32686 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.25.109.62:33903,127.25.109.61:43277'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 16:02:21.576879 32686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:21.577958 32686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:21.587103 32694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:21.587128 32686 server_base.cc:1034] running on GCE node
W20260924 16:02:21.587150 32691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:21.587123 32692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:21.588017 32686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:21.589236 32686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:21.590458 32686 hybrid_clock.cc:648] HybridClock initialized: now 1790265741590439 us; error 35 us; skew 500 ppm
Sep 24 16:02:21 dist-test-slave-n106 krb5kdc[32176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265741, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:21.593492 32686 init.cc:377] Logged in from keytab as oryx/127.25.109.61@KRBTEST.COM (short username oryx)
I20260924 16:02:21.594507 32686 webserver.cc:460] Webserver started at http://127.25.109.61:33193/ using document root <none> and password file <none>
I20260924 16:02:21.595083 32686 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:21.595176 32686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:21.595424 32686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:21.597579 32686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-1/data/instance:
uuid: "daf9b928e16b41c787926bfe44bf616e"
format_stamp: "Formatted at 2026-09-24 16:02:21 on dist-test-slave-n106"
server_key: "55e3fbf29a03276e05df3db5ce9aad3e"
server_key_iv: "ad2e3606a1ae92aead6990d7180cfa6a"
server_key_version: "encryptionkey@0"
I20260924 16:02:21.598174 32686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-1/wal/instance:
uuid: "daf9b928e16b41c787926bfe44bf616e"
format_stamp: "Formatted at 2026-09-24 16:02:21 on dist-test-slave-n106"
server_key: "55e3fbf29a03276e05df3db5ce9aad3e"
server_key_iv: "ad2e3606a1ae92aead6990d7180cfa6a"
server_key_version: "encryptionkey@0"
I20260924 16:02:21.602110 32686 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:02:21.604990 32701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:21.606372 32686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:02:21.606506 32686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-1/wal
uuid: "daf9b928e16b41c787926bfe44bf616e"
format_stamp: "Formatted at 2026-09-24 16:02:21 on dist-test-slave-n106"
server_key: "55e3fbf29a03276e05df3db5ce9aad3e"
server_key_iv: "ad2e3606a1ae92aead6990d7180cfa6a"
server_key_version: "encryptionkey@0"
I20260924 16:02:21.606607 32686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:21.619099 32686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:21.622648 32686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:21.622805 32686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:21.630353 32686 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.61:43277
I20260924 16:02:21.630708 32753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.61:43277 every 8 connection(s)
I20260924 16:02:21.631394 32686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-1/data/info.pb
I20260924 16:02:21.634306 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 32686
I20260924 16:02:21.635006 32754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:21.640241 32754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:02:21 dist-test-slave-n106 krb5kdc[32176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.109.61@KRBTEST.COM
Sep 24 16:02:21 dist-test-slave-n106 krb5kdc[32176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265741, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.61@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM
I20260924 16:02:21.650333 32757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:21.635886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40376 (local address 127.25.109.61:43277)
0924 16:02:21.636341 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:21.636356 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:21.636390 (+ 34us) server_negotiation.cc:407] Connection header received
0924 16:02:21.637208 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:21.637227 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:21.637548 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:21.637975 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:21.638789 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.639785 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.640794 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.641277 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.641465 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:21.644693 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:21.644699 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:21.644708 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:21.644734 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:21.646649 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:21.647176 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:21.647183 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:21.647190 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:21.647322 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:21.647662 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:21.647667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:21.647669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:21.648126 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:21.648383 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:21.648718 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 16:02:21.649082 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":153,"threads_started":1}
I20260924 16:02:21.653160 32260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:21.642382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41078 (local address 127.25.109.62:33903)
0924 16:02:21.642524 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:21.642529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:21.643254 (+ 725us) server_negotiation.cc:407] Connection header received
0924 16:02:21.643521 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:21.643525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:21.643586 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:21.643686 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:21.644621 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.645520 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.646486 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.646729 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.646841 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:21.649275 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:21.649280 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:21.649282 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:21.649313 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:21.650952 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:21.651462 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:21.651465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:21.651467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:21.651565 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:21.651899 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:21.651902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:21.651904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:21.652212 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:21.652323 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:21.652813 (+ 490us) server_negotiation.cc:299] Negotiation successful
0924 16:02:21.652952 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 16:02:21.653172 32758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:21.642377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.0.0.1:41078)
0924 16:02:21.643129 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:21.643166 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:21.643314 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:21.643769 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:21.643778 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:21.644270 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:21.644481 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:21.644491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.645631 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.645635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:21.646350 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:21.646360 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.646846 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.646850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:21.646972 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:21.647495 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:21.647507 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:21.649119 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:21.651064 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:21.651071 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:21.651075 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:21.651355 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:21.651699 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:21.651702 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:21.651703 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:21.651790 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:21.652352 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:21.652357 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:21.652533 (+ 176us) client_negotiation.cc:769] Sending connection context
0924 16:02:21.652706 (+ 173us) client_negotiation.cc:240] Negotiation successful
0924 16:02:21.652897 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":92,"threads_started":1}
I20260924 16:02:21.654754 32754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:02:21 dist-test-slave-n106 krb5kdc[32176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265741, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.61@KRBTEST.COM for oryx/127.25.109.61@KRBTEST.COM
I20260924 16:02:21.662858 32758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:21.655291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:43277 (local address 127.0.0.1:40390)
0924 16:02:21.655408 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:21.655420 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:21.655522 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:21.655749 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:21.655752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:21.655922 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:21.656003 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:21.656008 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.657278 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.657281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:21.657773 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:21.657780 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.658231 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.658234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:21.658351 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:21.658778 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:21.658789 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:21.659989 (+ 1200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:21.661535 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:21.661537 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:21.661538 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:21.661728 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:21.661964 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:21.661966 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:21.661967 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:21.662040 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:21.662432 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:21.662434 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:21.662513 (+ 79us) client_negotiation.cc:769] Sending connection context
0924 16:02:21.662592 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 16:02:21.662692 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":4}
I20260924 16:02:21.662873 32757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:21.655410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40390 (local address 127.25.109.61:43277)
0924 16:02:21.655508 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:21.655538 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:21.655554 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:02:21.655591 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:21.655597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:21.655630 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:21.655724 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:21.656124 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.657140 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.657891 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.658126 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.658234 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:21.660118 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:21.660121 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:21.660123 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:21.660144 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:21.661436 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:21.661813 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:21.661816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:21.661817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:21.661885 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:21.662156 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:21.662159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:21.662160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:21.662345 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:21.662460 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:21.662653 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 16:02:21.662748 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":41}
I20260924 16:02:21.665984 32754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e: Bootstrap starting.
I20260924 16:02:21.668366 32754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:21.669179 32754 log.cc:824] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:21.671131 32754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e: No bootstrap required, opened a new log
I20260924 16:02:21.674268 32754 raft_consensus.cc:348] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } }
I20260924 16:02:21.674469 32754 raft_consensus.cc:374] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:21.674566 32754 raft_consensus.cc:729] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daf9b928e16b41c787926bfe44bf616e, State: Initialized, Role: FOLLOWER
I20260924 16:02:21.675040 32754 consensus_queue.cc:260] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } }
I20260924 16:02:21.675522 32760 sys_catalog.cc:455] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } } }
I20260924 16:02:21.675649 32760 sys_catalog.cc:458] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:02:21.676273 32754 sys_catalog.cc:564] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [sys.catalog]: configured and running, proceeding with master startup.
W20260924 16:02:21.681171 303 catalog_manager.cc:1558] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:02:21.681259 303 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:02:21.685489 32686 master_runner.cc:428] Detected that this master daf9b928e16b41c787926bfe44bf616e is joining an existing cluster
I20260924 16:02:21.685549 32686 master_runner.cc:432] Initiating AddMaster RPC to add 127.25.109.61:43277
I20260924 16:02:21.696529 32260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:21.688140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41092 (local address 127.25.109.62:33903)
0924 16:02:21.688285 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:21.688290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:21.688306 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:02:21.688351 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:21.688355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:21.688447 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:21.688530 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:21.689093 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.689882 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.690946 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.691275 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.691378 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:21.692988 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:21.692993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:21.692995 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:21.693029 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:21.694673 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:21.695182 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:21.695185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:21.695187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:21.695273 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:21.695575 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:21.695581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:21.695586 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:21.695796 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:21.695895 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:21.696197 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 16:02:21.696327 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 16:02:21.696626 309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:21.687692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:43277 (local address 127.0.0.1:40402)
0924 16:02:21.688163 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:21.688177 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:21.688307 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:21.688562 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:21.688566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:21.688866 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:21.689015 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:21.689026 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.690026 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.690030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:21.690987 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:21.690999 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.691633 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.691636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:21.691698 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:21.692188 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:21.692200 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:21.692673 (+ 473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:21.694778 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:21.694781 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:21.694783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:21.695129 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:21.695409 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:21.695412 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:21.695413 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:21.695485 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:21.695933 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:21.695936 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:21.696021 (+ 85us) client_negotiation.cc:769] Sending connection context
0924 16:02:21.696131 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 16:02:21.696439 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":149,"spinlock_wait_cycles":2048,"thread_start_us":109,"threads_started":1}
I20260924 16:02:21.697009 310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:21.687670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.0.0.1:41092)
0924 16:02:21.688105 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:21.688122 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:21.688217 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:21.688648 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:21.688651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:21.688861 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:21.688960 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:21.688965 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.690020 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.690023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:21.690818 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:21.690827 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.691415 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.691421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:21.691575 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:21.692344 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:21.692359 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:21.692840 (+ 481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:21.694783 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:21.694786 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:21.694788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:21.695077 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:21.695372 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:21.695375 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:21.695377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:21.695469 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:21.695895 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:21.695898 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:21.696007 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 16:02:21.696136 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 16:02:21.696334 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":70,"spinlock_wait_cycles":13440,"thread_start_us":113,"threads_started":1}
I20260924 16:02:21.697026 32757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:21.687749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40402 (local address 127.25.109.61:43277)
0924 16:02:21.687865 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:21.687869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:21.688243 (+ 374us) server_negotiation.cc:407] Connection header received
0924 16:02:21.688398 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:21.688420 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:21.688456 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:21.688528 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:21.689113 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.689881 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.691105 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.691422 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.691512 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:21.692799 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:21.692802 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:21.692805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:21.692826 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:21.694645 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:21.695242 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:21.695245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:21.695246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:21.695317 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:21.695596 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:21.695598 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:21.695600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:21.695796 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:21.695872 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:21.696830 (+ 958us) server_negotiation.cc:299] Negotiation successful
0924 16:02:21.696938 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":40}
Sep 24 16:02:21 dist-test-slave-n106 krb5kdc[32176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265740, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.62@KRBTEST.COM for oryx/127.25.109.61@KRBTEST.COM
I20260924 16:02:21.713982 311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:21.701542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:43277 (local address 127.0.0.1:40406)
0924 16:02:21.702394 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:21.702453 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:21.702596 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:21.702973 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:21.702980 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:21.703521 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:21.703679 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:21.703687 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.704641 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.704645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:21.705884 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:21.705896 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.706486 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.706492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:21.706651 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:21.707288 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:21.707304 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:21.709067 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:21.711590 (+ 2523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:21.711605 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:21.711615 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:21.711980 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:21.712353 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:21.712356 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:21.712358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:21.712492 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:21.713142 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:21.713150 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:21.713312 (+ 162us) client_negotiation.cc:769] Sending connection context
0924 16:02:21.713506 (+ 194us) client_negotiation.cc:240] Negotiation successful
0924 16:02:21.713746 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":98,"threads_started":1}
I20260924 16:02:21.714769 32757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:21.701604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40406 (local address 127.25.109.61:43277)
0924 16:02:21.701716 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:21.701719 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:21.702560 (+ 841us) server_negotiation.cc:407] Connection header received
0924 16:02:21.702778 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:21.702781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:21.702830 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:21.702907 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:21.703801 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.704515 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.706017 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.706348 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.706448 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:21.709234 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:21.709241 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:21.709247 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:21.709282 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:21.711395 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:21.712119 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:21.712126 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:21.712129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:21.712230 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:21.712609 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:21.712614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:21.712617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:21.712968 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:21.713073 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:21.714438 (+ 1365us) server_negotiation.cc:299] Negotiation successful
0924 16:02:21.714588 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for oryx/127.25.109.60@KRBTEST.COM; defaulting to no policy
I20260924 16:02:21.716843 32221 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.25.109.61:43277: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "daf9b928e16b41c787926bfe44bf616e"
member_type: NON_VOTER
last_known_addr {
host: "127.25.109.61"
port: 43277
}
attrs {
promote: true
}
}
dest_uuid: "bda7fa2c9ca5483ea36450964edfe158"
cas_config_opid_index: -1
I20260924 16:02:21.717600 32221 consensus_queue.cc:237] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: NON_VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.109.60@KRBTEST.COM" created.
I20260924 16:02:21.719766 315 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER daf9b928e16b41c787926bfe44bf616e (127.25.109.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: NON_VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: true } } }
I20260924 16:02:21.720352 32728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:21.721755 315 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.25.109.61:43277
I20260924 16:02:21.722221 32728 raft_consensus.cc:1261] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [term 1 FOLLOWER]: Refusing update from remote peer bda7fa2c9ca5483ea36450964edfe158: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 16:02:21.722268 32686 master.cc:537] Master@127.25.109.61:43277 shutting down...
I20260924 16:02:21.722414 317 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bda7fa2c9ca5483ea36450964edfe158" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: NON_VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: true } } }
I20260924 16:02:21.722615 317 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:21.723181 314 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [LEADER]: Connected to new peer: Peer: permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: NON_VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:21.726912 32686 raft_consensus.cc:2229] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:21.727226 32686 raft_consensus.cc:2258] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:21.727367 32686 tablet_replica.cc:330] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e: stopping tablet replica
W20260924 16:02:21.727499 32206 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260924 16:02:21.728487 32206 consensus_peers.cc:487] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 -> Peer daf9b928e16b41c787926bfe44bf616e (127.25.109.61:43277): Couldn't send request to peer daf9b928e16b41c787926bfe44bf616e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.109.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.60.keytab.
Entry for principal oryx/127.25.109.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.60.keytab.
I20260924 16:02:21.740162 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.60:44917
--webserver_interface=127.25.109.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.60.keytab
--principal=oryx/127.25.109.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:33903,127.25.109.61:43277,127.25.109.60:44917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:02:21.742655 32686 master.cc:552] Master@127.25.109.61:43277 shutdown complete.
I20260924 16:02:21.742741 32686 master_runner.cc:305] Clearing existing system tablet
I20260924 16:02:21.743047 32686 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:21.743144 32686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:21.744344 32686 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 16:02:21.745313 322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:21.745534 32686 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:21.745669 32686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 16:02:21.745707 32686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-1/wal
uuid: "daf9b928e16b41c787926bfe44bf616e"
format_stamp: "Formatted at 2026-09-24 16:02:21 on dist-test-slave-n106"
server_key: "55e3fbf29a03276e05df3db5ce9aad3e"
server_key_iv: "ad2e3606a1ae92aead6990d7180cfa6a"
server_key_version: "encryptionkey@0"
I20260924 16:02:21.747534 32686 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:21.749951 32686 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 16:02:21.750075 32686 log.cc:1196] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 16:02:21.750360 32686 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e: Deleting consensus metadata
I20260924 16:02:21.750579 32686 master_runner.cc:315] Copying system tablet from 127.25.109.62:33903
I20260924 16:02:21.751854 32686 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.62:33903
I20260924 16:02:21.760205 32260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:21.752654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41106 (local address 127.25.109.62:33903)
0924 16:02:21.752792 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:21.752796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:21.753041 (+ 245us) server_negotiation.cc:407] Connection header received
0924 16:02:21.753198 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:21.753202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:21.753308 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:21.753381 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:21.753869 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.754411 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.755363 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.755608 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.755723 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:21.756976 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:21.756980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:21.756983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:21.757011 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:21.758509 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:21.758953 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:21.758957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:21.758959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:21.759045 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:21.759328 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:21.759332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:21.759334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:21.759529 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:21.759637 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:21.759917 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 16:02:21.760035 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:02:21.760215 328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:21.752599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.0.0.1:41106)
0924 16:02:21.752933 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:21.752945 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:21.753093 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:21.753401 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:21.753404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:21.753627 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:21.753738 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:21.753744 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.754529 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.754535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:21.755237 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:21.755246 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.755730 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.755734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:21.755831 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:21.756475 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:21.756496 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:21.756827 (+ 331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:21.758632 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:21.758637 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:21.758639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:21.758851 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:21.759140 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:21.759144 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:21.759146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:21.759236 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:21.759627 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:21.759630 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:21.759740 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 16:02:21.759848 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 16:02:21.760001 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":92,"threads_started":1}
I20260924 16:02:21.761132 32241 tablet_copy_service.cc:140] P bda7fa2c9ca5483ea36450964edfe158: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer daf9b928e16b41c787926bfe44bf616e ({username='oryx', principal='oryx/127.25.109.61@KRBTEST.COM'} at 127.0.0.1:41106)
I20260924 16:02:21.761332 32241 tablet_copy_service.cc:161] P bda7fa2c9ca5483ea36450964edfe158: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer daf9b928e16b41c787926bfe44bf616e at {username='oryx', principal='oryx/127.25.109.61@KRBTEST.COM'} at 127.0.0.1:41106: session id = daf9b928e16b41c787926bfe44bf616e-00000000000000000000000000000000
I20260924 16:02:21.762828 32241 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 16:02:21.764547 32686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 16:02:21.766580 32686 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e: tablet copy: Starting download of 0 data blocks...
I20260924 16:02:21.766798 32686 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e: tablet copy: Starting download of 1 WAL segments...
I20260924 16:02:21.768560 32686 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 16:02:21.770290 32241 tablet_copy_service.cc:342] P bda7fa2c9ca5483ea36450964edfe158: Request end of tablet copy session daf9b928e16b41c787926bfe44bf616e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.25.109.61@KRBTEST.COM'} at 127.0.0.1:41106
I20260924 16:02:21.770424 32241 tablet_copy_service.cc:433] P bda7fa2c9ca5483ea36450964edfe158: ending tablet copy session daf9b928e16b41c787926bfe44bf616e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer daf9b928e16b41c787926bfe44bf616e
I20260924 16:02:21.778126 32686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:21.779892 332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:21.780061 329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:21.780365 32686 server_base.cc:1034] running on GCE node
W20260924 16:02:21.784605 330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:21.784876 32686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:21.785228 32686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:21.786365 32686 hybrid_clock.cc:648] HybridClock initialized: now 1790265741786333 us; error 53 us; skew 500 ppm
Sep 24 16:02:21 dist-test-slave-n106 krb5kdc[32176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265741, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:21.787946 32686 init.cc:377] Logged in from keytab as oryx/127.25.109.61@KRBTEST.COM (short username oryx)
I20260924 16:02:21.788487 32686 webserver.cc:460] Webserver started at http://127.25.109.61:37839/ using document root <none> and password file <none>
I20260924 16:02:21.788725 32686 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:21.788780 32686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:21.789937 32686 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 16:02:21.790967 339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:21.791203 32686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 16:02:21.791301 32686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-1/wal
uuid: "daf9b928e16b41c787926bfe44bf616e"
format_stamp: "Formatted at 2026-09-24 16:02:21 on dist-test-slave-n106"
server_key: "55e3fbf29a03276e05df3db5ce9aad3e"
server_key_iv: "ad2e3606a1ae92aead6990d7180cfa6a"
server_key_version: "encryptionkey@0"
I20260924 16:02:21.791409 32686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:21.813758 32686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:21.815977 32686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:21.821484 32686 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.61:43277
I20260924 16:02:21.821666 391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.61:43277 every 8 connection(s)
I20260924 16:02:21.822116 32686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-1/data/info.pb
I20260924 16:02:21.823679 392 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:02:21.824635 392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e: Bootstrap starting.
I20260924 16:02:21.833387 392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:02:21.833735 392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e: Bootstrap complete.
I20260924 16:02:21.834240 392 raft_consensus.cc:348] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: NON_VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: true } }
I20260924 16:02:21.834388 392 raft_consensus.cc:729] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: daf9b928e16b41c787926bfe44bf616e, State: Initialized, Role: LEARNER
I20260924 16:02:21.834507 392 consensus_queue.cc:260] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: NON_VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: true } }
I20260924 16:02:21.834836 395 sys_catalog.cc:455] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: NON_VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: true } } }
I20260924 16:02:21.834990 395 sys_catalog.cc:458] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [sys.catalog]: This master's current role is: LEARNER
I20260924 16:02:21.834998 392 sys_catalog.cc:564] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:02:21.838639 406 catalog_manager.cc:1259] Loaded cluster ID: 3cb794533a7b41a6955bf42add1fa079
I20260924 16:02:21.838716 406 catalog_manager.cc:1552] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e: loading cluster ID for follower catalog manager: success
I20260924 16:02:21.840250 406 catalog_manager.cc:1574] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e: acquiring CA information for follower catalog manager: success
I20260924 16:02:21.841400 406 catalog_manager.cc:1602] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 16:02:21.898145 320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:21.898439 320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:21.898486 320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:21.902714 320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:02:21.902781 320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:21.902802 320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:21.902818 320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:02:21.902834 320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:02:21.908314 320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:33903,127.25.109.61:43277,127.25.109.60:44917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.60.keytab
--principal=oryx/127.25.109.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.60:44917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.109.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:21.909592 320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:21.910686 320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:21.919697 411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:21.919704 414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:21.919745 320 server_base.cc:1034] running on GCE node
W20260924 16:02:21.919708 412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:21.920636 320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:21.921710 320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:21.922899 320 hybrid_clock.cc:648] HybridClock initialized: now 1790265741922882 us; error 35 us; skew 500 ppm
Sep 24 16:02:21 dist-test-slave-n106 krb5kdc[32176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265741, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:21.925850 320 init.cc:377] Logged in from keytab as oryx/127.25.109.60@KRBTEST.COM (short username oryx)
I20260924 16:02:21.926736 320 webserver.cc:460] Webserver started at http://127.25.109.60:43209/ using document root <none> and password file <none>
I20260924 16:02:21.927294 320 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:21.927403 320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:21.927651 320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:21.929730 320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-2/data/instance:
uuid: "9450614f7a7c4bcebf504f2db67ba64e"
format_stamp: "Formatted at 2026-09-24 16:02:21 on dist-test-slave-n106"
server_key: "6965a2c8c244af04fea0a8ecc5fc13f5"
server_key_iv: "de7ed47c94eb546a92f9c36f14eee010"
server_key_version: "encryptionkey@0"
I20260924 16:02:21.930366 320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-2/wal/instance:
uuid: "9450614f7a7c4bcebf504f2db67ba64e"
format_stamp: "Formatted at 2026-09-24 16:02:21 on dist-test-slave-n106"
server_key: "6965a2c8c244af04fea0a8ecc5fc13f5"
server_key_iv: "de7ed47c94eb546a92f9c36f14eee010"
server_key_version: "encryptionkey@0"
I20260924 16:02:21.934369 320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:02:21.937268 421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:21.938560 320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:02:21.938697 320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-2/wal
uuid: "9450614f7a7c4bcebf504f2db67ba64e"
format_stamp: "Formatted at 2026-09-24 16:02:21 on dist-test-slave-n106"
server_key: "6965a2c8c244af04fea0a8ecc5fc13f5"
server_key_iv: "de7ed47c94eb546a92f9c36f14eee010"
server_key_version: "encryptionkey@0"
I20260924 16:02:21.938822 320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:21.952972 320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:21.954675 32413 heartbeater.cc:503] Master 127.25.109.62:33903 was elected leader, sending a full tablet report...
I20260924 16:02:21.956595 320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:21.956743 320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:21.963027 320 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.60:44917
I20260924 16:02:21.963308 473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.60:44917 every 8 connection(s)
I20260924 16:02:21.964039 320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-2/data/info.pb
I20260924 16:02:21.966832 474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:21.968113 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 320
I20260924 16:02:21.971191 474 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:02:21 dist-test-slave-n106 krb5kdc[32176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.109.60@KRBTEST.COM
Sep 24 16:02:21 dist-test-slave-n106 krb5kdc[32176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265741, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.60@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM
I20260924 16:02:21.983139 476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:21.969625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49338 (local address 127.25.109.60:44917)
0924 16:02:21.970060 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:21.970078 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:21.970121 (+ 43us) server_negotiation.cc:407] Connection header received
0924 16:02:21.970974 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:21.970997 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:21.971405 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:21.971809 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:21.972472 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.973793 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.974612 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.975064 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.975245 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:21.977912 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:21.977917 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:21.977927 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:21.977949 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:21.979747 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:21.980322 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:21.980329 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:21.980334 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:21.980505 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:21.980861 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:21.980865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:21.980866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:21.981396 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:21.981620 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:21.981877 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 16:02:21.982153 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":124,"threads_started":1}
I20260924 16:02:21.983356 32260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:21.973184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41110 (local address 127.25.109.62:33903)
0924 16:02:21.973330 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:21.973335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:21.973581 (+ 246us) server_negotiation.cc:407] Connection header received
0924 16:02:21.973744 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:21.973747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:21.973795 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:21.973890 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:21.975266 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.975777 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.976742 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.976957 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.977061 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:21.979704 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:21.979709 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:21.979712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:21.979739 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:21.981516 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:21.982076 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:21.982079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:21.982081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:21.982169 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:21.982429 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:21.982432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:21.982434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:21.982648 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:21.982748 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:21.983084 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 16:02:21.983203 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:02:21.983425 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:21.972976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.0.0.1:41110)
0924 16:02:21.973455 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:21.973502 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:21.973646 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:21.974690 (+ 1044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:21.974695 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:21.975004 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:21.975131 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:21.975137 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.975912 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.975920 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:21.976613 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:21.976623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.977111 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.977115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:21.977214 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:21.977764 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:21.977779 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:21.979547 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:21.981639 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:21.981648 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:21.981654 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:21.981953 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:21.982262 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:21.982264 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:21.982266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:21.982333 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:21.982753 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:21.982758 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:21.982850 (+ 92us) client_negotiation.cc:769] Sending connection context
0924 16:02:21.983004 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 16:02:21.983182 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":104,"threads_started":1}
I20260924 16:02:21.985105 474 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:02:21 dist-test-slave-n106 krb5kdc[32176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265741, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.60@KRBTEST.COM for oryx/127.25.109.61@KRBTEST.COM
I20260924 16:02:21.993001 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:21.985559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:43277 (local address 127.0.0.1:40410)
0924 16:02:21.985679 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:21.985692 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:21.985766 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:21.986171 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:21.986174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:21.986291 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:21.986375 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:21.986379 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.987258 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.987260 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:21.987759 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:21.987766 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.988266 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.988270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:21.988370 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:21.988857 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:21.988869 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:21.989931 (+ 1062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:21.991684 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:21.991686 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:21.991688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:21.991865 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:21.992126 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:21.992128 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:21.992129 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:21.992201 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:21.992590 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:21.992592 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:21.992663 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 16:02:21.992799 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 16:02:21.992894 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 16:02:21.993139 478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:21.985701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40410 (local address 127.25.109.61:43277)
0924 16:02:21.985960 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:21.985971 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:21.985983 (+ 12us) server_negotiation.cc:407] Connection header received
0924 16:02:21.986027 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:21.986029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:21.986066 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:21.986136 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:21.986474 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.987165 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.987896 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.988144 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.988257 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:21.990074 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:21.990077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:21.990079 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:21.990103 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:21.991534 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:21.991949 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:21.991952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:21.991953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:21.992029 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:21.992278 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:21.992280 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:21.992281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:21.992508 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:21.992587 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:21.992870 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 16:02:21.992969 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":97,"threads_started":1}
I20260924 16:02:21.993819 474 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.60" port: 44917 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:02:21 dist-test-slave-n106 krb5kdc[32176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265741, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.60@KRBTEST.COM for oryx/127.25.109.60@KRBTEST.COM
I20260924 16:02:22.001134 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:21.994206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:44917 (local address 127.0.0.1:49344)
0924 16:02:21.994312 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:21.994327 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:21.994393 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:21.994558 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:21.994560 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:21.994761 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:21.994829 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:21.994834 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.995824 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.995827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:21.996297 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:21.996304 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.996762 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.996765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:21.996821 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:21.997219 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:21.997231 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:21.998377 (+ 1146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:21.999946 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:21.999949 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:21.999950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:22.000120 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:22.000348 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:22.000350 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:22.000351 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:22.000478 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:22.000775 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:22.000778 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:22.000846 (+ 68us) client_negotiation.cc:769] Sending connection context
0924 16:02:22.000922 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 16:02:22.001007 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 16:02:22.001240 476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:21.994298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49344 (local address 127.25.109.60:44917)
0924 16:02:21.994400 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:21.994404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:21.994419 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:02:21.994456 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:21.994458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:21.994489 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:21.994549 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:21.994925 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.995729 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.996391 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:21.996667 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:21.996744 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:21.998496 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:21.998499 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:21.998501 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:21.998521 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:21.999855 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.000199 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.000201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.000203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.000271 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.000543 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.000545 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.000546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.000696 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:22.000771 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:22.000978 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 16:02:22.001059 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 16:02:22.004037 474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e: Bootstrap starting.
I20260924 16:02:22.006407 474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:22.007131 474 log.cc:824] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:22.008911 474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e: No bootstrap required, opened a new log
I20260924 16:02:22.011730 474 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 44917 } }
I20260924 16:02:22.011950 474 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:22.012034 474 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9450614f7a7c4bcebf504f2db67ba64e, State: Initialized, Role: FOLLOWER
I20260924 16:02:22.012532 474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 44917 } }
I20260924 16:02:22.012964 480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 44917 } } }
I20260924 16:02:22.013073 480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:02:22.013685 474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [sys.catalog]: configured and running, proceeding with master startup.
W20260924 16:02:22.018416 491 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:02:22.018505 491 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:02:22.021888 320 master_runner.cc:428] Detected that this master 9450614f7a7c4bcebf504f2db67ba64e is joining an existing cluster
I20260924 16:02:22.021957 320 master_runner.cc:432] Initiating AddMaster RPC to add 127.25.109.60:44917
I20260924 16:02:22.036718 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 32284
I20260924 16:02:22.043833 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:46785
--local_ip_for_outbound_sockets=127.25.109.1
--tserver_master_addrs=127.25.109.62:33903
--webserver_port=43059
--webserver_interface=127.25.109.1
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.1.keytab
--principal=oryx/127.25.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.25.109.62:33903,127.25.109.61:43277
--tserver_master_addrs=127.25.109.62:33903,127.25.109.61:43277,127.25.109.60:44917 with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:02:22.054699 498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.024047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:43277 (local address 127.0.0.1:40416)
0924 16:02:22.024651 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:22.024668 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:22.024729 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:22.025222 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:22.025227 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:22.025437 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:22.025545 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.025553 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.026573 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.026577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.027479 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.027491 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.028111 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.028115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.028240 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.028834 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:22.028855 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:22.049321 (+ 20466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:22.051786 (+ 2465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:22.051789 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:22.051792 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:22.052048 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:22.052454 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:22.052458 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:22.052460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:22.052582 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:22.053754 (+ 1172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:22.053757 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:22.054136 (+ 379us) client_negotiation.cc:769] Sending connection context
0924 16:02:22.054258 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 16:02:22.054414 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":1,"spinlock_wait_cycles":1792,"thread_start_us":99,"threads_started":1}
I20260924 16:02:22.055382 478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.024474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40416 (local address 127.25.109.61:43277)
0924 16:02:22.024975 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:22.024980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:22.024993 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:02:22.025031 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:22.025034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:22.025080 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:22.025183 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:22.025685 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.026444 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.027628 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.027985 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.028105 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.049573 (+ 21468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:22.049579 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:22.049584 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:22.049624 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:22.051670 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.052217 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.052221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.052223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.052323 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.052969 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.052973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.052975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.053264 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:22.053715 (+ 451us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:22.055069 (+ 1354us) server_negotiation.cc:299] Negotiation successful
0924 16:02:22.055216 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416}
I20260924 16:02:22.058044 32260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.024437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41124 (local address 127.25.109.62:33903)
0924 16:02:22.024573 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:22.024578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:22.024593 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:02:22.024650 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:22.024653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:22.024704 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:22.024793 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:22.025326 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.026174 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.027859 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.028167 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.028271 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.052976 (+ 24705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:22.052980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:22.052983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:22.053033 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:22.054773 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.055294 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.055298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.055300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.055397 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.056951 (+ 1554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.056955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.056958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.057239 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:22.057320 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:22.057704 (+ 384us) server_negotiation.cc:299] Negotiation successful
0924 16:02:22.057838 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:02:22.058674 497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.024025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.0.0.1:41124)
0924 16:02:22.024333 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:22.024349 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:22.024484 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:22.024827 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:22.024830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:22.025024 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:22.025180 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.025189 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.026313 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.026316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.027745 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.027752 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.028289 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.028293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.028398 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.029291 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:22.029306 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:22.052783 (+ 23477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:22.054912 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:22.054915 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:22.054917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:22.055171 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:22.056716 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:22.056721 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:22.056726 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:22.056824 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:22.057400 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:22.057403 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:22.057504 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 16:02:22.058312 (+ 808us) client_negotiation.cc:240] Negotiation successful
0924 16:02:22.058485 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":112,"threads_started":1}
I20260924 16:02:22.059212 476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.025476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49360 (local address 127.25.109.60:44917)
0924 16:02:22.025573 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:22.025578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:22.028725 (+ 3147us) server_negotiation.cc:407] Connection header received
0924 16:02:22.028869 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:22.028874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:22.028912 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:22.028955 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:22.049623 (+ 20668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.050461 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.051783 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.052144 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.052216 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.055917 (+ 3701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:22.055920 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:22.055923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:22.055943 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:22.057646 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.058061 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.058066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.058067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.058144 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.058412 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.058415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.058416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.058607 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:22.058691 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:22.058984 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 16:02:22.059088 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 16:02:22.059664 499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.025747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:44917 (local address 127.0.0.1:49360)
0924 16:02:22.028633 (+ 2886us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:22.028644 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:22.028779 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:22.029095 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:22.029098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:22.049303 (+ 20205us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:22.049508 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.049517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.050645 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.050650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.051649 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.051661 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.052381 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.052387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.052516 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.053487 (+ 971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:22.053503 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:22.055791 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:22.057748 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:22.057750 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:22.057752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:22.057964 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:22.058244 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:22.058247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:22.058248 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:22.058330 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:22.058739 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:22.058743 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:22.058827 (+ 84us) client_negotiation.cc:769] Sending connection context
0924 16:02:22.059384 (+ 557us) client_negotiation.cc:240] Negotiation successful
0924 16:02:22.059502 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2827,"spinlock_wait_cycles":2304,"thread_start_us":87,"threads_started":1}
Sep 24 16:02:22 dist-test-slave-n106 krb5kdc[32176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265740, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.62@KRBTEST.COM for oryx/127.25.109.60@KRBTEST.COM
I20260924 16:02:22.071412 476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.062964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49362 (local address 127.25.109.60:44917)
0924 16:02:22.063279 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:22.063282 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:22.063292 (+ 10us) server_negotiation.cc:407] Connection header received
0924 16:02:22.063345 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:22.063347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:22.063384 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:22.063488 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:22.063885 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.064366 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.065341 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.065573 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.065681 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.068280 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:22.068284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:22.068287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:22.068321 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:22.069798 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.070232 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.070237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.070239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.070320 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.070600 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.070605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.070607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.070779 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:22.070882 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:22.071148 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 16:02:22.071249 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259}
I20260924 16:02:22.071412 311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.062911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:44917 (local address 127.0.0.1:49362)
0924 16:02:22.063093 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:22.063105 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:22.063168 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:22.063507 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:22.063518 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:22.063693 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:22.063775 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.063780 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.064683 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.064685 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.065219 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.065227 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.065688 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.065692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.065790 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.066236 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:22.066250 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:22.068130 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:22.069915 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:22.069920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:22.069922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:22.070131 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:22.070415 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:22.070418 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:22.070420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:22.070508 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:22.070876 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:22.070879 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:22.070982 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 16:02:22.071097 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 16:02:22.071249 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260924 16:02:22.072943 32221 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.25.109.60:44917: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e"
member_type: NON_VOTER
last_known_addr {
host: "127.25.109.60"
port: 44917
}
attrs {
promote: true
}
}
dest_uuid: "bda7fa2c9ca5483ea36450964edfe158"
cas_config_opid_index: 5
I20260924 16:02:22.073565 32221 consensus_queue.cc:237] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: NON_VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: true } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: NON_VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: true } }
I20260924 16:02:22.074754 315 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9450614f7a7c4bcebf504f2db67ba64e (127.25.109.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: NON_VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: true } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: NON_VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: true } } }
I20260924 16:02:22.075399 448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:22.076283 315 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.25.109.60:44917
I20260924 16:02:22.076498 315 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bda7fa2c9ca5483ea36450964edfe158" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: NON_VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: true } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: NON_VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: true } } }
I20260924 16:02:22.076594 315 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:22.076748 448 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [term 1 FOLLOWER]: Refusing update from remote peer bda7fa2c9ca5483ea36450964edfe158: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 16:02:22.077214 320 master.cc:537] Master@127.25.109.60:44917 shutting down...
I20260924 16:02:22.077656 317 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: NON_VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 16:02:22.079739 32207 consensus_peers.cc:487] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 -> Peer 9450614f7a7c4bcebf504f2db67ba64e (127.25.109.60:44917): Couldn't send request to peer 9450614f7a7c4bcebf504f2db67ba64e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 16:02:22.083547 311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.074913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:43277 (local address 127.0.0.1:40418)
0924 16:02:22.077295 (+ 2382us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:22.077311 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:22.077917 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:22.078407 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:22.078410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:22.078431 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:22.078649 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.078660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.079724 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.079728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.081318 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.081331 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.082985 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.082991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.083090 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.083149 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 16:02:22.083234 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 16:02:22.083339 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2281}
I20260924 16:02:22.083954 478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.076771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40418 (local address 127.25.109.61:43277)
0924 16:02:22.076932 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:22.076936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:22.077839 (+ 903us) server_negotiation.cc:407] Connection header received
0924 16:02:22.078077 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:22.078080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:22.078148 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:22.078241 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:22.078812 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.079589 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.081500 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.082218 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.082329 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.082920 (+ 591us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:22.083729 (+ 809us) server_negotiation.cc:299] Negotiation successful
0924 16:02:22.083835 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:02:22.084508 366 raft_consensus.cc:1261] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [term 1 LEARNER]: Refusing update from remote peer bda7fa2c9ca5483ea36450964edfe158: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 16:02:22.085088 317 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [LEADER]: Connected to new peer: Peer: permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: NON_VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 16:02:22.085929 320 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:22.086241 320 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:22.086383 320 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e: stopping tablet replica
I20260924 16:02:22.087494 366 raft_consensus.cc:2932] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9450614f7a7c4bcebf504f2db67ba64e (127.25.109.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: NON_VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: true } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: NON_VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: true } } }
I20260924 16:02:22.089813 395 sys_catalog.cc:455] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bda7fa2c9ca5483ea36450964edfe158. Latest consensus state: current_term: 1 leader_uuid: "bda7fa2c9ca5483ea36450964edfe158" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: NON_VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: true } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: NON_VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: true } } }
I20260924 16:02:22.089908 395 sys_catalog.cc:458] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [sys.catalog]: This master's current role is: LEARNER
I20260924 16:02:22.089978 395 sys_catalog.cc:455] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bda7fa2c9ca5483ea36450964edfe158" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: NON_VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: true } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: NON_VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: true } } }
I20260924 16:02:22.090029 395 sys_catalog.cc:458] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [sys.catalog]: This master's current role is: LEARNER
I20260924 16:02:22.091686 314 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158: attempting to promote NON_VOTER daf9b928e16b41c787926bfe44bf616e to VOTER
I20260924 16:02:22.092271 314 consensus_queue.cc:237] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: NON_VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: true } }
I20260924 16:02:22.094087 366 raft_consensus.cc:1261] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [term 1 LEARNER]: Refusing update from remote peer bda7fa2c9ca5483ea36450964edfe158: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260924 16:02:22.094149 32207 consensus_peers.cc:487] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 -> Peer 9450614f7a7c4bcebf504f2db67ba64e (127.25.109.60:44917): Couldn't send request to peer 9450614f7a7c4bcebf504f2db67ba64e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 16:02:22.094558 317 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [LEADER]: Connected to new peer: Peer: permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 16:02:22.096448 315 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, daf9b928e16b41c787926bfe44bf616e (127.25.109.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: NON_VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: true } } }
I20260924 16:02:22.097750 315 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bda7fa2c9ca5483ea36450964edfe158" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: NON_VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: true } } }
I20260924 16:02:22.097859 315 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:22.097997 315 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bda7fa2c9ca5483ea36450964edfe158" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: NON_VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: true } } }
I20260924 16:02:22.098112 315 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:22.098644 366 raft_consensus.cc:2932] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, daf9b928e16b41c787926bfe44bf616e (127.25.109.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: NON_VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: true } } }
I20260924 16:02:22.099978 502 sys_catalog.cc:455] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bda7fa2c9ca5483ea36450964edfe158" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: NON_VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: true } } }
I20260924 16:02:22.100127 502 sys_catalog.cc:458] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:02:22.106738 320 master.cc:552] Master@127.25.109.60:44917 shutdown complete.
I20260924 16:02:22.106815 320 master_runner.cc:305] Clearing existing system tablet
I20260924 16:02:22.107054 320 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:22.107126 320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:22.108239 320 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 16:02:22.109196 504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:22.109416 320 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:22.109573 320 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 16:02:22.109611 320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-2/wal
uuid: "9450614f7a7c4bcebf504f2db67ba64e"
format_stamp: "Formatted at 2026-09-24 16:02:21 on dist-test-slave-n106"
server_key: "6965a2c8c244af04fea0a8ecc5fc13f5"
server_key_iv: "de7ed47c94eb546a92f9c36f14eee010"
server_key_version: "encryptionkey@0"
I20260924 16:02:22.111403 320 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:22.113427 320 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 16:02:22.113566 320 log.cc:1196] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 16:02:22.113835 320 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e: Deleting consensus metadata
I20260924 16:02:22.114027 320 master_runner.cc:315] Copying system tablet from 127.25.109.62:33903
I20260924 16:02:22.115242 320 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.62:33903
I20260924 16:02:22.123896 32260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.116244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41140 (local address 127.25.109.62:33903)
0924 16:02:22.116537 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:22.116542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:22.116686 (+ 144us) server_negotiation.cc:407] Connection header received
0924 16:02:22.116852 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:22.116855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:22.116912 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:22.116995 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:22.117448 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.117994 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.118860 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.119160 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.119258 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.120430 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:22.120435 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:22.120438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:22.120465 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:22.122087 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.122588 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.122591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.122593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.122678 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.122990 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.122996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.122999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.123199 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:22.123312 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:22.123614 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 16:02:22.123727 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260924 16:02:22.123903 510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.116182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.0.0.1:41140)
0924 16:02:22.116593 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:22.116606 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:22.116746 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:22.117014 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:22.117017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:22.117218 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:22.117325 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.117331 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.118128 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.118134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.118730 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.118738 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.119271 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.119275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.119375 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.119937 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:22.119949 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:22.120266 (+ 317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:22.122216 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:22.122219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:22.122221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:22.122478 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:22.122779 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:22.122782 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:22.122784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:22.122892 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:22.123320 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:22.123325 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:22.123437 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 16:02:22.123557 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 16:02:22.123712 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":81,"threads_started":1}
I20260924 16:02:22.124682 32241 tablet_copy_service.cc:140] P bda7fa2c9ca5483ea36450964edfe158: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9450614f7a7c4bcebf504f2db67ba64e ({username='oryx', principal='oryx/127.25.109.60@KRBTEST.COM'} at 127.0.0.1:41140)
I20260924 16:02:22.124799 32241 tablet_copy_service.cc:161] P bda7fa2c9ca5483ea36450964edfe158: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9450614f7a7c4bcebf504f2db67ba64e at {username='oryx', principal='oryx/127.25.109.60@KRBTEST.COM'} at 127.0.0.1:41140: session id = 9450614f7a7c4bcebf504f2db67ba64e-00000000000000000000000000000000
I20260924 16:02:22.125850 32241 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 16:02:22.127058 320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 16:02:22.129045 320 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e: tablet copy: Starting download of 0 data blocks...
I20260924 16:02:22.129290 320 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e: tablet copy: Starting download of 1 WAL segments...
I20260924 16:02:22.130689 320 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 16:02:22.132298 32241 tablet_copy_service.cc:342] P bda7fa2c9ca5483ea36450964edfe158: Request end of tablet copy session 9450614f7a7c4bcebf504f2db67ba64e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.25.109.60@KRBTEST.COM'} at 127.0.0.1:41140
I20260924 16:02:22.132407 32241 tablet_copy_service.cc:433] P bda7fa2c9ca5483ea36450964edfe158: ending tablet copy session 9450614f7a7c4bcebf504f2db67ba64e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9450614f7a7c4bcebf504f2db67ba64e
I20260924 16:02:22.140128 320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:22.141932 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:22.141959 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:22.142239 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:22.144485 320 server_base.cc:1034] running on GCE node
I20260924 16:02:22.144677 320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:22.145030 320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:22.146188 320 hybrid_clock.cc:648] HybridClock initialized: now 1790265742146172 us; error 38 us; skew 500 ppm
Sep 24 16:02:22 dist-test-slave-n106 krb5kdc[32176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265742, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:22.147708 320 init.cc:377] Logged in from keytab as oryx/127.25.109.60@KRBTEST.COM (short username oryx)
I20260924 16:02:22.148227 320 webserver.cc:460] Webserver started at http://127.25.109.60:41811/ using document root <none> and password file <none>
I20260924 16:02:22.148548 320 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:22.148607 320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:22.150076 320 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 16:02:22.151090 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:22.151328 320 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 16:02:22.151427 320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-2/wal
uuid: "9450614f7a7c4bcebf504f2db67ba64e"
format_stamp: "Formatted at 2026-09-24 16:02:21 on dist-test-slave-n106"
server_key: "6965a2c8c244af04fea0a8ecc5fc13f5"
server_key_iv: "de7ed47c94eb546a92f9c36f14eee010"
server_key_version: "encryptionkey@0"
I20260924 16:02:22.151551 320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:22.161525 320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:22.163945 320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:22.168975 320 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.60:44917
I20260924 16:02:22.168987 573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.60:44917 every 8 connection(s)
I20260924 16:02:22.169585 320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/master-2/data/info.pb
I20260924 16:02:22.171198 574 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:02:22.172070 574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e: Bootstrap starting.
I20260924 16:02:22.181550 574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:02:22.181878 574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e: Bootstrap complete.
I20260924 16:02:22.182528 32547 heartbeater.cc:503] Master 127.25.109.62:33903 was elected leader, sending a full tablet report...
I20260924 16:02:22.182484 574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: NON_VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: true } }
I20260924 16:02:22.182626 574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9450614f7a7c4bcebf504f2db67ba64e, State: Initialized, Role: LEARNER
I20260924 16:02:22.182732 574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: NON_VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: true } }
I20260924 16:02:22.183245 574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:02:22.184432 577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: NON_VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: true } } }
I20260924 16:02:22.184866 577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [sys.catalog]: This master's current role is: LEARNER
I20260924 16:02:22.186977 588 catalog_manager.cc:1259] Loaded cluster ID: 3cb794533a7b41a6955bf42add1fa079
I20260924 16:02:22.187060 588 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e: loading cluster ID for follower catalog manager: success
I20260924 16:02:22.188753 588 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e: acquiring CA information for follower catalog manager: success
I20260924 16:02:22.189980 588 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 16:02:22.209985 500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:22.210395 500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:22.210460 500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:22.216857 500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:22.216953 500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:22.217054 500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:02:22.225826 500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.1.keytab
--principal=oryx/127.25.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:46785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=43059
--tserver_master_addrs=127.25.109.62:33903,127.25.109.61:43277,127.25.109.60:44917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:22.227294 500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:22.228802 500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:02:22.240332 500 server_base.cc:1034] running on GCE node
W20260924 16:02:22.240296 596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:22.240279 594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:22.240279 593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:22.241008 500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:22.241719 500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:22.242931 500 hybrid_clock.cc:648] HybridClock initialized: now 1790265742242912 us; error 35 us; skew 500 ppm
Sep 24 16:02:22 dist-test-slave-n106 krb5kdc[32176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265742, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:22.245985 500 init.cc:377] Logged in from keytab as oryx/127.25.109.1@KRBTEST.COM (short username oryx)
I20260924 16:02:22.246907 500 webserver.cc:460] Webserver started at http://127.25.109.1:43059/ using document root <none> and password file <none>
I20260924 16:02:22.247463 500 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:22.247542 500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:22.252149 500 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:02:22.254727 603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:22.256101 500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:02:22.256244 500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "c3fa4c2ab3d94a049605e22b6492d8ab"
format_stamp: "Formatted at 2026-09-24 16:02:20 on dist-test-slave-n106"
server_key: "48254c4cc748111afcf2d4f4deafce3d"
server_key_iv: "37d3d00ebb8afb5c3cd5933745f66246"
server_key_version: "encryptionkey@0"
I20260924 16:02:22.256907 500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:22.285820 500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:22.289410 500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:22.289639 500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:22.290292 500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:22.291319 500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:22.291400 500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:22.291471 500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:22.291527 500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:22.301049 500 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:46785
I20260924 16:02:22.301079 717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:46785 every 8 connection(s)
I20260924 16:02:22.302088 500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:02:22.302856 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 500
I20260924 16:02:22.303057 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 32418
I20260924 16:02:22.311272 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:38957
--local_ip_for_outbound_sockets=127.25.109.2
--tserver_master_addrs=127.25.109.62:33903
--webserver_port=33049
--webserver_interface=127.25.109.2
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.2.keytab
--principal=oryx/127.25.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.25.109.62:33903,127.25.109.61:43277
--tserver_master_addrs=127.25.109.62:33903,127.25.109.61:43277,127.25.109.60:44917 with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:02:22 dist-test-slave-n106 krb5kdc[32176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265742, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.60@KRBTEST.COM
Sep 24 16:02:22 dist-test-slave-n106 krb5kdc[32176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265742, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM
Sep 24 16:02:22 dist-test-slave-n106 krb5kdc[32176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265742, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.61@KRBTEST.COM
I20260924 16:02:22.320938 724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.305941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:37257 (local address 127.25.109.60:44917)
0924 16:02:22.306408 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:22.306414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:22.306432 (+ 18us) server_negotiation.cc:407] Connection header received
0924 16:02:22.306952 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:22.306955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:22.307021 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:22.307526 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:22.308668 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.309746 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.311779 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.312204 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.312621 (+ 417us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.316145 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:22.316149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:22.316151 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:22.316179 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:22.318174 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.318815 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.318819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.318821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.318915 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.319426 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.319431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.319435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.319700 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:22.319781 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:22.320580 (+ 799us) server_negotiation.cc:299] Negotiation successful
0924 16:02:22.320732 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":106,"threads_started":1}
I20260924 16:02:22.322352 32260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.304089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:34189 (local address 127.25.109.62:33903)
0924 16:02:22.304227 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:22.304231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:22.305273 (+ 1042us) server_negotiation.cc:407] Connection header received
0924 16:02:22.306883 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:22.306887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:22.306953 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:22.307009 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:22.309297 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.310182 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.311598 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.311933 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.312017 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.317906 (+ 5889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:22.317913 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:22.317916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:22.317972 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:22.320592 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.321046 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.321049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.321051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.321141 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.321448 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.321451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.321453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.321669 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:22.321758 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:22.322092 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 16:02:22.322213 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 16:02:22.323122 722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.304586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.25.109.1:34189)
0924 16:02:22.305111 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:22.305155 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:22.306617 (+ 1462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:22.307341 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:22.307350 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:22.308983 (+ 1633us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:22.309149 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.309159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.310344 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.310350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.311451 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.311463 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.312229 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.312236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.312559 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.313621 (+ 1062us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:22.313642 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:22.317732 (+ 4090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:22.320705 (+ 2973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:22.320708 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:22.320711 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:22.320926 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:22.321267 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:22.321270 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:22.321272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:22.321351 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:22.321815 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:22.321818 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:22.321897 (+ 79us) client_negotiation.cc:769] Sending connection context
0924 16:02:22.322575 (+ 678us) client_negotiation.cc:240] Negotiation successful
0924 16:02:22.322711 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":122,"spinlock_wait_cycles":4352,"thread_start_us":150,"threads_started":1}
I20260924 16:02:22.323140 478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.308257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:60971 (local address 127.25.109.61:43277)
0924 16:02:22.308760 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:22.308764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:22.308780 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:02:22.308823 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:22.308825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:22.308876 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:22.308955 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:22.309533 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.310300 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.311740 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.312068 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.312144 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.319104 (+ 6960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:22.319108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:22.319112 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:22.319151 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:22.320923 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.321484 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.321491 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.321495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.321591 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.321963 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.321969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.321973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.322236 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:22.322322 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:22.322756 (+ 434us) server_negotiation.cc:299] Negotiation successful
0924 16:02:22.322907 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404}
I20260924 16:02:22.323786 725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.308130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:43277 (local address 127.25.109.1:60971)
0924 16:02:22.308617 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:22.308634 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:22.308726 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:22.309073 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:22.309076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:22.309266 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:22.309380 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.309387 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.310473 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.310476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.311624 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.311633 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.312287 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.312291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.312892 (+ 601us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.316169 (+ 3277us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:22.316182 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:22.318941 (+ 2759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:22.321064 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:22.321069 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:22.321075 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:22.321351 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:22.321739 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:22.321744 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:22.321749 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:22.321849 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:22.322411 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:22.322415 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:22.322526 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 16:02:22.323300 (+ 774us) client_negotiation.cc:240] Negotiation successful
0924 16:02:22.323508 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":125,"thread_start_us":122,"threads_started":1}
I20260924 16:02:22.324306 723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.305790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:44917 (local address 127.25.109.1:37257)
0924 16:02:22.306140 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:22.306156 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:22.306617 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:22.307341 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:22.307350 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:22.307843 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:22.308514 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.308531 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.309904 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.309907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.311632 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.311646 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.312435 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.312443 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.312584 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.313386 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:22.313413 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:22.315959 (+ 2546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:22.318309 (+ 2350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:22.318318 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:22.318338 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:22.318689 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:22.319060 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:22.319064 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:22.319065 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:22.319300 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:22.319878 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:22.319884 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:22.320250 (+ 366us) client_negotiation.cc:769] Sending connection context
0924 16:02:22.321430 (+ 1180us) client_negotiation.cc:240] Negotiation successful
0924 16:02:22.321792 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":260,"spinlock_wait_cycles":1280,"thread_start_us":91,"threads_started":1}
I20260924 16:02:22.324594 719 heartbeater.cc:347] Connected to a master server at 127.25.109.62:33903
I20260924 16:02:22.324853 719 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:22.325153 720 heartbeater.cc:347] Connected to a master server at 127.25.109.61:43277
I20260924 16:02:22.325240 720 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:22.325482 719 heartbeater.cc:511] Master 127.25.109.62:33903 requested a full tablet report, sending...
I20260924 16:02:22.325482 720 heartbeater.cc:511] Master 127.25.109.61:43277 requested a full tablet report, sending...
I20260924 16:02:22.325747 718 heartbeater.cc:347] Connected to a master server at 127.25.109.60:44917
I20260924 16:02:22.325814 718 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:22.325950 718 heartbeater.cc:511] Master 127.25.109.60:44917 requested a full tablet report, sending...
I20260924 16:02:22.327265 356 ts_manager.cc:194] Registered new tserver with Master: c3fa4c2ab3d94a049605e22b6492d8ab (127.25.109.1:46785)
I20260924 16:02:22.327265 538 ts_manager.cc:194] Registered new tserver with Master: c3fa4c2ab3d94a049605e22b6492d8ab (127.25.109.1:46785)
I20260924 16:02:22.328364 32221 ts_manager.cc:194] Re-registered known tserver with Master: c3fa4c2ab3d94a049605e22b6492d8ab (127.25.109.1:46785)
I20260924 16:02:22.328994 32221 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.109.1@KRBTEST.COM'} at 127.25.109.1:34189
I20260924 16:02:22.409524 32681 heartbeater.cc:503] Master 127.25.109.62:33903 was elected leader, sending a full tablet report...
W20260924 16:02:22.442832 726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:22.443142 726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:22.443226 726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:22.447358 726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:22.447450 726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:22.447542 726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:02:22.453912 726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.2.keytab
--principal=oryx/127.25.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:38957
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=33049
--tserver_master_addrs=127.25.109.62:33903,127.25.109.61:43277,127.25.109.60:44917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:22.455152 726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:22.456298 726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:22.466774 731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:22.466796 732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:22.466782 734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:22.466984 726 server_base.cc:1034] running on GCE node
I20260924 16:02:22.467551 726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:22.468235 726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:22.469434 726 hybrid_clock.cc:648] HybridClock initialized: now 1790265742469402 us; error 53 us; skew 500 ppm
Sep 24 16:02:22 dist-test-slave-n106 krb5kdc[32176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265742, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:22.472345 726 init.cc:377] Logged in from keytab as oryx/127.25.109.2@KRBTEST.COM (short username oryx)
I20260924 16:02:22.473256 726 webserver.cc:460] Webserver started at http://127.25.109.2:33049/ using document root <none> and password file <none>
I20260924 16:02:22.473793 726 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:22.473863 726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:22.477725 726 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:02:22.479830 741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:22.481159 726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:02:22.481297 726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "f758e6c10f3448cdb5d1e3ba31d30925"
format_stamp: "Formatted at 2026-09-24 16:02:21 on dist-test-slave-n106"
server_key: "cc027d32ee6955225589ca1c58e845d9"
server_key_iv: "6f1d3fd423142ffd5acea9fd58bf281a"
server_key_version: "encryptionkey@0"
I20260924 16:02:22.481856 726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:22.521116 311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.517315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:44917 (local address 127.0.0.1:49376)
0924 16:02:22.517487 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:22.517504 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:22.517621 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:22.517901 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:22.517903 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:22.517917 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:22.518006 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.518010 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.519020 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.519023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.519815 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.519825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.520767 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.520770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.520841 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.520864 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 16:02:22.520917 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 16:02:22.520982 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 16:02:22.521173 724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.517438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49376 (local address 127.25.109.60:44917)
0924 16:02:22.517609 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:22.517614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:22.517634 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:02:22.517702 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:22.517705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:22.517773 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:22.517879 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:22.518122 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.518910 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.519947 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.520652 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.520756 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.520945 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:22.520976 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 16:02:22.521045 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 16:02:22.522013 548 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 16:02:22.522424 577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bda7fa2c9ca5483ea36450964edfe158. Latest consensus state: current_term: 1 leader_uuid: "bda7fa2c9ca5483ea36450964edfe158" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: NON_VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: true } } }
I20260924 16:02:22.522620 577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [sys.catalog]: This master's current role is: LEARNER
I20260924 16:02:22.527174 726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:22.530786 726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:22.531003 726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:22.531672 726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:22.532655 726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:22.532706 726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:22.532776 726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:22.532807 726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:22.542367 726 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:38957
I20260924 16:02:22.542445 854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:38957 every 8 connection(s)
I20260924 16:02:22.543499 726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:02:22.552561 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 726
I20260924 16:02:22.552794 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 32552
Sep 24 16:02:22 dist-test-slave-n106 krb5kdc[32176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265742, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM
Sep 24 16:02:22 dist-test-slave-n106 krb5kdc[32176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265742, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.61@KRBTEST.COM
Sep 24 16:02:22 dist-test-slave-n106 krb5kdc[32176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265742, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.60@KRBTEST.COM
I20260924 16:02:22.561230 32260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.545453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:43575 (local address 127.25.109.62:33903)
0924 16:02:22.545597 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:22.545602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:22.547430 (+ 1828us) server_negotiation.cc:407] Connection header received
0924 16:02:22.547866 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:22.547870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:22.547926 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:22.547981 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:22.549591 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.550411 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.551840 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.552118 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.552220 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.556383 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:22.556387 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:22.556390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:22.556459 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:22.558745 (+ 2286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.559341 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.559345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.559347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.559562 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.559877 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.559880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.559882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.560102 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:22.560174 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:22.560928 (+ 754us) server_negotiation.cc:299] Negotiation successful
0924 16:02:22.561063 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:02:22.561230 724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.545449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:35241 (local address 127.25.109.60:44917)
0924 16:02:22.545592 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:22.545597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:22.547243 (+ 1646us) server_negotiation.cc:407] Connection header received
0924 16:02:22.547855 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:22.547858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:22.547921 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:22.548001 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:22.549631 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.550256 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.551608 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.551969 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.552071 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.556837 (+ 4766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:22.556844 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:22.556850 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:22.556887 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:22.558675 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.559341 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.559345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.559347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.559446 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.559883 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.559886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.559889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.560099 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:22.560174 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:22.560928 (+ 754us) server_negotiation.cc:299] Negotiation successful
0924 16:02:22.561059 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:02:22.561334 478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.545939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:37205 (local address 127.25.109.61:43277)
0924 16:02:22.546059 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:22.546064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:22.546709 (+ 645us) server_negotiation.cc:407] Connection header received
0924 16:02:22.547865 (+ 1156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:22.547869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:22.547926 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:22.547981 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:22.549591 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.550408 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.551804 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.552159 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.552265 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.556151 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:22.556158 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:22.556164 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:22.556205 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:22.558471 (+ 2266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.559245 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.559252 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.559257 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.559349 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.559761 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.559767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.559772 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.560030 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:22.560122 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:22.560931 (+ 809us) server_negotiation.cc:299] Negotiation successful
0924 16:02:22.561078 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 16:02:22.563051 859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.545961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:43277 (local address 127.25.109.2:37205)
0924 16:02:22.546564 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:22.546608 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:22.547624 (+ 1016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:22.548198 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:22.548207 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:22.548751 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:22.549422 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.549431 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.550557 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.550560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.551692 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.551703 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.552440 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.552445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.552596 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.553439 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:22.553470 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:22.555889 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:22.558613 (+ 2724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:22.558622 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:22.558643 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:22.559123 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:22.559462 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:22.559468 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:22.559472 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:22.559654 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:22.560213 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:22.560222 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:22.560617 (+ 395us) client_negotiation.cc:769] Sending connection context
0924 16:02:22.562211 (+ 1594us) client_negotiation.cc:240] Negotiation successful
0924 16:02:22.562387 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":296,"spinlock_wait_cycles":110720,"thread_start_us":165,"threads_started":1}
I20260924 16:02:22.563048 861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.545961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.25.109.2:43575)
0924 16:02:22.547349 (+ 1388us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:22.547358 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:22.547624 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:22.548198 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:22.548207 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:22.548751 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:22.549419 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.549431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.550557 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.550560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.551692 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.551703 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.552345 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.552349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.552471 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.553481 (+ 1010us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:22.553493 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:22.556244 (+ 2751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:22.558856 (+ 2612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:22.558861 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:22.558865 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:22.559221 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:22.559656 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:22.559663 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:22.559668 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:22.559767 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:22.560253 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:22.560256 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:22.560615 (+ 359us) client_negotiation.cc:769] Sending connection context
0924 16:02:22.561414 (+ 799us) client_negotiation.cc:240] Negotiation successful
0924 16:02:22.561928 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1318,"mutex_wait_us":625,"spinlock_wait_cycles":1399296,"thread_start_us":728,"threads_started":1}
I20260924 16:02:22.563372 860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.545961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:44917 (local address 127.25.109.2:35241)
0924 16:02:22.547141 (+ 1180us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:22.547153 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:22.547625 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:22.548197 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:22.548205 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:22.548719 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:22.549412 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.549430 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.550454 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.550457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.551471 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.551481 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.552127 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.552131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.552334 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.553446 (+ 1112us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:22.553474 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:22.556627 (+ 3153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:22.558809 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:22.558814 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:22.558816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:22.559221 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:22.559706 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:22.559712 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:22.559716 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:22.559782 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:22.560306 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:22.560309 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:22.560615 (+ 306us) client_negotiation.cc:769] Sending connection context
0924 16:02:22.561615 (+ 1000us) client_negotiation.cc:240] Negotiation successful
0924 16:02:22.561881 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1081,"mutex_wait_us":74,"spinlock_wait_cycles":68352,"thread_start_us":734,"threads_started":1}
I20260924 16:02:22.564623 857 heartbeater.cc:347] Connected to a master server at 127.25.109.61:43277
I20260924 16:02:22.564927 857 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:22.565161 855 heartbeater.cc:347] Connected to a master server at 127.25.109.60:44917
I20260924 16:02:22.565168 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:41657
--local_ip_for_outbound_sockets=127.25.109.3
--tserver_master_addrs=127.25.109.62:33903
--webserver_port=38853
--webserver_interface=127.25.109.3
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.3.keytab
--principal=oryx/127.25.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.25.109.62:33903,127.25.109.61:43277
--tserver_master_addrs=127.25.109.62:33903,127.25.109.61:43277,127.25.109.60:44917 with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:02:22.565266 855 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:22.565582 855 heartbeater.cc:511] Master 127.25.109.60:44917 requested a full tablet report, sending...
I20260924 16:02:22.565585 857 heartbeater.cc:511] Master 127.25.109.61:43277 requested a full tablet report, sending...
I20260924 16:02:22.564644 856 heartbeater.cc:347] Connected to a master server at 127.25.109.62:33903
I20260924 16:02:22.566860 856 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:22.567058 856 heartbeater.cc:511] Master 127.25.109.62:33903 requested a full tablet report, sending...
I20260924 16:02:22.568156 32221 ts_manager.cc:194] Re-registered known tserver with Master: f758e6c10f3448cdb5d1e3ba31d30925 (127.25.109.2:38957)
I20260924 16:02:22.568396 538 ts_manager.cc:194] Registered new tserver with Master: f758e6c10f3448cdb5d1e3ba31d30925 (127.25.109.2:38957)
I20260924 16:02:22.568953 32221 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.109.2@KRBTEST.COM'} at 127.25.109.2:43575
I20260924 16:02:22.569430 356 ts_manager.cc:194] Registered new tserver with Master: f758e6c10f3448cdb5d1e3ba31d30925 (127.25.109.2:38957)
W20260924 16:02:22.683652 862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:22.683913 862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:22.683957 862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:22.688272 862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:22.688335 862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:22.688395 862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:02:22.694061 862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.3.keytab
--principal=oryx/127.25.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:41657
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=38853
--tserver_master_addrs=127.25.109.62:33903,127.25.109.61:43277,127.25.109.60:44917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:22.695148 862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:22.696261 862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:22.706504 870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:22.706559 868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:22.706578 867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:22.706763 862 server_base.cc:1034] running on GCE node
I20260924 16:02:22.707170 862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:22.707842 862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:22.709031 862 hybrid_clock.cc:648] HybridClock initialized: now 1790265742709018 us; error 26 us; skew 500 ppm
Sep 24 16:02:22 dist-test-slave-n106 krb5kdc[32176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265742, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:22.711891 862 init.cc:377] Logged in from keytab as oryx/127.25.109.3@KRBTEST.COM (short username oryx)
I20260924 16:02:22.712918 862 webserver.cc:460] Webserver started at http://127.25.109.3:38853/ using document root <none> and password file <none>
I20260924 16:02:22.713510 862 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:22.713639 862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:22.717710 862 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260924 16:02:22.719821 877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:22.721264 862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:02:22.721417 862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "1c9c5af925794483afe9f26d1a0f4291"
format_stamp: "Formatted at 2026-09-24 16:02:21 on dist-test-slave-n106"
server_key: "bca5284f64488c93897c1eae1ce90559"
server_key_iv: "9aeb2ecd0612215437eb508a503ec12f"
server_key_version: "encryptionkey@0"
I20260924 16:02:22.722007 862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:22.736768 862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:22.739956 862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:22.740120 862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:22.740715 862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:22.741572 862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:22.741628 862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:22.741672 862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:22.741688 862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:22.751346 862 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:41657
I20260924 16:02:22.751433 990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:41657 every 8 connection(s)
I20260924 16:02:22.752573 862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:02:22.752934 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 862
Sep 24 16:02:22 dist-test-slave-n106 krb5kdc[32176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265742, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.60@KRBTEST.COM
Sep 24 16:02:22 dist-test-slave-n106 krb5kdc[32176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265742, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.61@KRBTEST.COM
I20260924 16:02:22.769868 32260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.755420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41156 (local address 127.25.109.62:33903)
0924 16:02:22.755556 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:22.755560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:22.757531 (+ 1971us) server_negotiation.cc:407] Connection header received
0924 16:02:22.757759 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:22.757763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:22.757824 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:22.757880 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:22.759029 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.759921 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.761280 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.761621 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.761703 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.764026 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:22.764030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:22.764033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:22.764068 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:22.766746 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.767791 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.767796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.767799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.767892 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.768329 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.768346 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.768349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.768905 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:22.769043 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:22.769535 (+ 492us) server_negotiation.cc:299] Negotiation successful
0924 16:02:22.769682 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 16:02:22.769964 1004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.756482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49380 (local address 127.25.109.60:44917)
0924 16:02:22.756752 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:22.756757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:22.757606 (+ 849us) server_negotiation.cc:407] Connection header received
0924 16:02:22.757830 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:22.757834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:22.757888 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:22.757974 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:22.759051 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.759825 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.761324 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.761654 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.761725 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.763932 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:22.763936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:22.763938 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:22.763966 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:22.766915 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.768544 (+ 1629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.768548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.768550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.768638 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.768988 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.768991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.768993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.769231 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:22.769344 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:22.769683 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 16:02:22.769804 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":95,"threads_started":1}
Sep 24 16:02:22 dist-test-slave-n106 krb5kdc[32176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265742, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM
I20260924 16:02:22.776710 478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.755606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:43285 (local address 127.25.109.61:43277)
0924 16:02:22.755734 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:22.755738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:22.756887 (+ 1149us) server_negotiation.cc:407] Connection header received
0924 16:02:22.758048 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:22.758051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:22.758110 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:22.758163 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:22.760053 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.760910 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.763657 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.764015 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.764123 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.771345 (+ 7222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:22.771351 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:22.771354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:22.771380 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:22.774941 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.775406 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.775410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.775411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.775489 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.775794 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.775798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.775799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.775990 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:22.776060 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:22.776391 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 16:02:22.776545 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":55}
I20260924 16:02:22.776970 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.756056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40426 (local address 127.25.109.61:43277)
0924 16:02:22.758939 (+ 2883us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:22.758945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:22.758959 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:02:22.759002 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:22.759006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:22.759049 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:22.759142 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:22.760066 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.762665 (+ 2599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.764075 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.764582 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.764677 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.767172 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:22.767176 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:22.767179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:22.767206 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:22.774839 (+ 7633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.775322 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.775326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.775327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.775416 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.775717 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.775720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.775721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.775940 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:22.776050 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:22.776686 (+ 636us) server_negotiation.cc:299] Negotiation successful
0924 16:02:22.776808 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":121,"server-negotiator.queue_time_us":2796,"thread_start_us":56,"threads_started":1}
I20260924 16:02:22.777557 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.755420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:39497 (local address 127.25.109.62:33903)
0924 16:02:22.764021 (+ 8601us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:22.764027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:22.764367 (+ 340us) server_negotiation.cc:407] Connection header received
0924 16:02:22.764558 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:22.764563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:22.764626 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:22.764721 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:22.765343 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.766143 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.767557 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.767938 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.768044 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.771458 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:22.771462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:22.771464 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:22.771487 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:22.774238 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.774765 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.774770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.774773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.774872 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.775319 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.775325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.775330 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.775571 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:22.775676 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:22.777267 (+ 1591us) server_negotiation.cc:299] Negotiation successful
0924 16:02:22.777390 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":148,"server-negotiator.queue_time_us":8511,"thread_start_us":70,"threads_started":1}
I20260924 16:02:22.774621 32221 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 16:02:22.778308 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.755294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.25.109.3:39497)
0924 16:02:22.764282 (+ 8988us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:22.764298 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:22.764427 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:22.764867 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:22.764871 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:22.765073 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:22.765214 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.765223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.766297 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.766301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.767412 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.767427 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.768178 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.768184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.768398 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.768963 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:22.768979 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:22.771335 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:22.774359 (+ 3024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:22.774364 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:22.774366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:22.774632 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:22.774983 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:22.774990 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:22.774993 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:22.775195 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:22.775730 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:22.775741 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:22.776102 (+ 361us) client_negotiation.cc:769] Sending connection context
0924 16:02:22.776327 (+ 225us) client_negotiation.cc:240] Negotiation successful
0924 16:02:22.776718 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8869,"mutex_wait_us":425,"thread_start_us":108,"threads_started":1}
I20260924 16:02:22.779019 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.755294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:43277 (local address 127.25.109.3:43285)
0924 16:02:22.756701 (+ 1407us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:22.756760 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:22.757820 (+ 1060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:22.758605 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:22.758615 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:22.759179 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:22.759825 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.759844 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.761143 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.761147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.762614 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.762640 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.764568 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.764572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.764875 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.766143 (+ 1268us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:22.767376 (+ 1233us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:22.771186 (+ 3810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:22.775052 (+ 3866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:22.775054 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:22.775056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:22.775297 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:22.775601 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:22.775604 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:22.775606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:22.775693 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:22.776144 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:22.776146 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:22.776220 (+ 74us) client_negotiation.cc:769] Sending connection context
0924 16:02:22.776926 (+ 706us) client_negotiation.cc:240] Negotiation successful
0924 16:02:22.777064 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1156,"mutex_wait_us":2,"spinlock_wait_cycles":2304,"thread_start_us":710,"threads_started":1}
W20260924 16:02:22.779312 32221 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:02:22.779793 993 heartbeater.cc:347] Connected to a master server at 127.25.109.61:43277
I20260924 16:02:22.779847 992 heartbeater.cc:347] Connected to a master server at 127.25.109.62:33903
I20260924 16:02:22.780009 993 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:22.780010 992 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:22.780740 992 heartbeater.cc:511] Master 127.25.109.62:33903 requested a full tablet report, sending...
I20260924 16:02:22.780743 993 heartbeater.cc:511] Master 127.25.109.61:43277 requested a full tablet report, sending...
I20260924 16:02:22.782033 356 ts_manager.cc:194] Registered new tserver with Master: 1c9c5af925794483afe9f26d1a0f4291 (127.25.109.3:41657)
I20260924 16:02:22.782040 32220 ts_manager.cc:194] Re-registered known tserver with Master: 1c9c5af925794483afe9f26d1a0f4291 (127.25.109.3:41657)
I20260924 16:02:22.782641 314 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bda7fa2c9ca5483ea36450964edfe158" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: NON_VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: true } } }
I20260924 16:02:22.782763 314 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:22.782867 32220 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.109.3@KRBTEST.COM'} at 127.25.109.3:39497
I20260924 16:02:22.783227 724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.754882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:60129 (local address 127.25.109.60:44917)
0924 16:02:22.755019 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:22.755024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:22.756847 (+ 1823us) server_negotiation.cc:407] Connection header received
0924 16:02:22.758042 (+ 1195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:22.758045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:22.758093 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:22.758161 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:22.760588 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.761309 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.765207 (+ 3898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.765538 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.765617 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.771234 (+ 5617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:22.771269 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:22.771274 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:22.771300 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:22.773114 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.773831 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.773835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.773836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.773921 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:22.781691 (+ 7770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:22.781695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:22.781697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:22.781968 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:22.782132 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:22.782916 (+ 784us) server_negotiation.cc:299] Negotiation successful
0924 16:02:22.783070 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 16:02:22.783890 999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.755294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:44917 (local address 127.25.109.3:60129)
0924 16:02:22.756700 (+ 1406us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:22.756760 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:22.757811 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:22.758604 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:22.758614 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:22.760284 (+ 1670us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:22.760437 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.760444 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.761467 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.761470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.763487 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.763502 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.766734 (+ 3232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.766753 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.766887 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.767539 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:22.767554 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:22.771053 (+ 3499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:22.773257 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:22.773264 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:22.773286 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:22.773702 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:22.774055 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:22.774059 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:22.774062 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:22.781554 (+ 7492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:22.782619 (+ 1065us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:22.782621 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:22.782713 (+ 92us) client_negotiation.cc:769] Sending connection context
0924 16:02:22.783527 (+ 814us) client_negotiation.cc:240] Negotiation successful
0924 16:02:22.783710 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1179,"mutex_wait_us":929,"spinlock_wait_cycles":94976,"thread_start_us":184,"threads_started":1}
I20260924 16:02:22.784643 991 heartbeater.cc:347] Connected to a master server at 127.25.109.60:44917
I20260924 16:02:22.784737 991 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:22.784936 991 heartbeater.cc:511] Master 127.25.109.60:44917 requested a full tablet report, sending...
I20260924 16:02:22.788764 314 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158: attempting to promote NON_VOTER 9450614f7a7c4bcebf504f2db67ba64e to VOTER
I20260924 16:02:22.789116 314 consensus_queue.cc:237] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: false } }
I20260924 16:02:22.789324 538 ts_manager.cc:194] Registered new tserver with Master: 1c9c5af925794483afe9f26d1a0f4291 (127.25.109.3:41657)
I20260924 16:02:22.790323 366 raft_consensus.cc:1261] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [term 1 FOLLOWER]: Refusing update from remote peer bda7fa2c9ca5483ea36450964edfe158: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 16:02:22.790710 1010 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [LEADER]: Connected to new peer: Peer: permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 16:02:22.790956 547 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [term 1 LEARNER]: Refusing update from remote peer bda7fa2c9ca5483ea36450964edfe158: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 16:02:22.791536 1010 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 16:02:22.793237 1010 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 9450614f7a7c4bcebf504f2db67ba64e (127.25.109.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: false } } }
I20260924 16:02:22.794831 1010 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bda7fa2c9ca5483ea36450964edfe158" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: false } } }
I20260924 16:02:22.794947 1010 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:22.795065 1010 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bda7fa2c9ca5483ea36450964edfe158" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: false } } }
I20260924 16:02:22.795169 1010 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:22.795274 1010 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bda7fa2c9ca5483ea36450964edfe158" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: false } } }
I20260924 16:02:22.795365 1010 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:22.795356 547 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 9450614f7a7c4bcebf504f2db67ba64e (127.25.109.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: false } } }
I20260924 16:02:22.795971 366 raft_consensus.cc:2932] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 9450614f7a7c4bcebf504f2db67ba64e (127.25.109.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: false } } }
I20260924 16:02:22.797776 577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bda7fa2c9ca5483ea36450964edfe158" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: false } } }
I20260924 16:02:22.797927 577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:02:22.799134 1021 sys_catalog.cc:455] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bda7fa2c9ca5483ea36450964edfe158" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: false } } }
I20260924 16:02:22.799275 1021 sys_catalog.cc:458] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:02:22.803498 1022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.796773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:38957 (local address 127.0.0.1:40346)
0924 16:02:22.798547 (+ 1774us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:22.798562 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:22.798654 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:22.799690 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:22.799693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:22.799712 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:22.799807 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.799812 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.801116 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.801119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.802231 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.802241 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.803074 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.803078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.803163 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.803187 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 16:02:22.803249 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 16:02:22.803325 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1706,"mutex_wait_us":22,"thread_start_us":121,"threads_started":1}
I20260924 16:02:22.803556 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.797243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40346 (local address 127.25.109.2:38957)
0924 16:02:22.799112 (+ 1869us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:22.799126 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:22.799153 (+ 27us) server_negotiation.cc:407] Connection header received
0924 16:02:22.799240 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:22.799246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:22.799518 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:22.799705 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:22.799941 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.801010 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.802361 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.802935 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.803062 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.803246 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:22.803300 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 16:02:22.803397 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1747,"thread_start_us":134,"threads_started":1}
I20260924 16:02:22.804575 311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.796526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:46785 (local address 127.0.0.1:40618)
0924 16:02:22.796834 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:22.796850 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:22.796930 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:22.800503 (+ 3573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:22.800506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:22.800526 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:22.800617 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.800622 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.801911 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.801915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.803005 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.803015 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.804172 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.804176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.804248 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.804267 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 16:02:22.804325 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 16:02:22.804402 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":1}
I20260924 16:02:22.804720 1019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.797131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40618 (local address 127.25.109.1:46785)
0924 16:02:22.800019 (+ 2888us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:22.800035 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:22.800055 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:02:22.800126 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:22.800131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:22.800321 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:22.800535 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:22.800718 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.801783 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.803152 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.804024 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.804125 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.804255 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:22.804353 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 16:02:22.804578 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2761,"thread_start_us":82,"threads_started":1}
I20260924 16:02:22.806892 789 tablet_service.cc:1467] Processing CreateTablet for tablet 1456c04181bd4d2eba410d0138021a9e (DEFAULT_TABLE table=test-table [id=a503e3bf0dff4294a1432e0934ab0c72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:22.807183 651 tablet_service.cc:1467] Processing CreateTablet for tablet 1456c04181bd4d2eba410d0138021a9e (DEFAULT_TABLE table=test-table [id=a503e3bf0dff4294a1432e0934ab0c72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:22.807971 789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1456c04181bd4d2eba410d0138021a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:22.808166 651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1456c04181bd4d2eba410d0138021a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:22.809212 1018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.798033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:41657 (local address 127.0.0.1:50188)
0924 16:02:22.798839 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:22.798852 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:22.798949 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:22.800013 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:22.800016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:22.800034 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:22.800112 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.800118 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.801459 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.801463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.802639 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.802650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.808822 (+ 6172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.808826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.808918 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.808945 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 16:02:22.808998 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 16:02:22.809073 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"thread_start_us":103,"threads_started":1}
I20260924 16:02:22.809885 1023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.798453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50188 (local address 127.25.109.3:41657)
0924 16:02:22.799530 (+ 1077us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:22.799543 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:22.799564 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:02:22.799640 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:22.799647 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:22.799838 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:22.800052 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:22.800193 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.801326 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.802795 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.808676 (+ 5881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.809413 (+ 737us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.809565 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:22.809639 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 16:02:22.809744 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":976,"thread_start_us":114,"threads_started":1}
I20260924 16:02:22.812824 925 tablet_service.cc:1467] Processing CreateTablet for tablet 1456c04181bd4d2eba410d0138021a9e (DEFAULT_TABLE table=test-table [id=a503e3bf0dff4294a1432e0934ab0c72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:22.813937 925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1456c04181bd4d2eba410d0138021a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:22.815407 1024 tablet_bootstrap.cc:492] T 1456c04181bd4d2eba410d0138021a9e P c3fa4c2ab3d94a049605e22b6492d8ab: Bootstrap starting.
I20260924 16:02:22.817233 1025 tablet_bootstrap.cc:492] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925: Bootstrap starting.
I20260924 16:02:22.817816 1024 tablet_bootstrap.cc:654] T 1456c04181bd4d2eba410d0138021a9e P c3fa4c2ab3d94a049605e22b6492d8ab: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:22.818580 1024 log.cc:824] T 1456c04181bd4d2eba410d0138021a9e P c3fa4c2ab3d94a049605e22b6492d8ab: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:22.819469 1025 tablet_bootstrap.cc:654] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:22.820339 1025 log.cc:824] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:22.820729 1024 tablet_bootstrap.cc:492] T 1456c04181bd4d2eba410d0138021a9e P c3fa4c2ab3d94a049605e22b6492d8ab: No bootstrap required, opened a new log
I20260924 16:02:22.820964 1024 ts_tablet_manager.cc:1397] T 1456c04181bd4d2eba410d0138021a9e P c3fa4c2ab3d94a049605e22b6492d8ab: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260924 16:02:22.822211 1028 tablet_bootstrap.cc:492] T 1456c04181bd4d2eba410d0138021a9e P 1c9c5af925794483afe9f26d1a0f4291: Bootstrap starting.
I20260924 16:02:22.822753 1025 tablet_bootstrap.cc:492] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925: No bootstrap required, opened a new log
I20260924 16:02:22.823020 1025 ts_tablet_manager.cc:1397] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 16:02:22.824954 1024 raft_consensus.cc:348] T 1456c04181bd4d2eba410d0138021a9e P c3fa4c2ab3d94a049605e22b6492d8ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3fa4c2ab3d94a049605e22b6492d8ab" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46785 } } peers { permanent_uuid: "f758e6c10f3448cdb5d1e3ba31d30925" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38957 } } peers { permanent_uuid: "1c9c5af925794483afe9f26d1a0f4291" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41657 } }
I20260924 16:02:22.825307 1024 raft_consensus.cc:374] T 1456c04181bd4d2eba410d0138021a9e P c3fa4c2ab3d94a049605e22b6492d8ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:22.825341 1028 tablet_bootstrap.cc:654] T 1456c04181bd4d2eba410d0138021a9e P 1c9c5af925794483afe9f26d1a0f4291: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:22.825390 1024 raft_consensus.cc:729] T 1456c04181bd4d2eba410d0138021a9e P c3fa4c2ab3d94a049605e22b6492d8ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3fa4c2ab3d94a049605e22b6492d8ab, State: Initialized, Role: FOLLOWER
I20260924 16:02:22.825902 1024 consensus_queue.cc:260] T 1456c04181bd4d2eba410d0138021a9e P c3fa4c2ab3d94a049605e22b6492d8ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3fa4c2ab3d94a049605e22b6492d8ab" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46785 } } peers { permanent_uuid: "f758e6c10f3448cdb5d1e3ba31d30925" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38957 } } peers { permanent_uuid: "1c9c5af925794483afe9f26d1a0f4291" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41657 } }
I20260924 16:02:22.826213 1028 log.cc:824] T 1456c04181bd4d2eba410d0138021a9e P 1c9c5af925794483afe9f26d1a0f4291: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:22.828079 1024 ts_tablet_manager.cc:1428] T 1456c04181bd4d2eba410d0138021a9e P c3fa4c2ab3d94a049605e22b6492d8ab: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 16:02:22.828365 1028 tablet_bootstrap.cc:492] T 1456c04181bd4d2eba410d0138021a9e P 1c9c5af925794483afe9f26d1a0f4291: No bootstrap required, opened a new log
I20260924 16:02:22.828474 719 heartbeater.cc:503] Master 127.25.109.62:33903 was elected leader, sending a full tablet report...
I20260924 16:02:22.828585 1028 ts_tablet_manager.cc:1397] T 1456c04181bd4d2eba410d0138021a9e P 1c9c5af925794483afe9f26d1a0f4291: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.002s
I20260924 16:02:22.832307 1025 raft_consensus.cc:348] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3fa4c2ab3d94a049605e22b6492d8ab" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46785 } } peers { permanent_uuid: "f758e6c10f3448cdb5d1e3ba31d30925" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38957 } } peers { permanent_uuid: "1c9c5af925794483afe9f26d1a0f4291" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41657 } }
I20260924 16:02:22.832698 1025 raft_consensus.cc:374] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:22.832778 1025 raft_consensus.cc:729] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f758e6c10f3448cdb5d1e3ba31d30925, State: Initialized, Role: FOLLOWER
I20260924 16:02:22.833258 1025 consensus_queue.cc:260] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3fa4c2ab3d94a049605e22b6492d8ab" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46785 } } peers { permanent_uuid: "f758e6c10f3448cdb5d1e3ba31d30925" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38957 } } peers { permanent_uuid: "1c9c5af925794483afe9f26d1a0f4291" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41657 } }
I20260924 16:02:22.834333 1025 ts_tablet_manager.cc:1428] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260924 16:02:22.834538 856 heartbeater.cc:503] Master 127.25.109.62:33903 was elected leader, sending a full tablet report...
I20260924 16:02:22.835605 1028 raft_consensus.cc:348] T 1456c04181bd4d2eba410d0138021a9e P 1c9c5af925794483afe9f26d1a0f4291 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3fa4c2ab3d94a049605e22b6492d8ab" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46785 } } peers { permanent_uuid: "f758e6c10f3448cdb5d1e3ba31d30925" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38957 } } peers { permanent_uuid: "1c9c5af925794483afe9f26d1a0f4291" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41657 } }
I20260924 16:02:22.835903 1028 raft_consensus.cc:374] T 1456c04181bd4d2eba410d0138021a9e P 1c9c5af925794483afe9f26d1a0f4291 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:22.835968 1028 raft_consensus.cc:729] T 1456c04181bd4d2eba410d0138021a9e P 1c9c5af925794483afe9f26d1a0f4291 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c9c5af925794483afe9f26d1a0f4291, State: Initialized, Role: FOLLOWER
I20260924 16:02:22.836577 1028 consensus_queue.cc:260] T 1456c04181bd4d2eba410d0138021a9e P 1c9c5af925794483afe9f26d1a0f4291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3fa4c2ab3d94a049605e22b6492d8ab" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46785 } } peers { permanent_uuid: "f758e6c10f3448cdb5d1e3ba31d30925" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38957 } } peers { permanent_uuid: "1c9c5af925794483afe9f26d1a0f4291" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41657 } }
I20260924 16:02:22.837833 1028 ts_tablet_manager.cc:1428] T 1456c04181bd4d2eba410d0138021a9e P 1c9c5af925794483afe9f26d1a0f4291: Time spent starting tablet: real 0.009s user 0.001s sys 0.004s
I20260924 16:02:22.838236 992 heartbeater.cc:503] Master 127.25.109.62:33903 was elected leader, sending a full tablet report...
I20260924 16:02:22.914901 1031 raft_consensus.cc:482] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:02:22.915172 1031 raft_consensus.cc:504] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3fa4c2ab3d94a049605e22b6492d8ab" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46785 } } peers { permanent_uuid: "f758e6c10f3448cdb5d1e3ba31d30925" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38957 } } peers { permanent_uuid: "1c9c5af925794483afe9f26d1a0f4291" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41657 } }
I20260924 16:02:22.916492 1031 leader_election.cc:288] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3fa4c2ab3d94a049605e22b6492d8ab (127.25.109.1:46785), 1c9c5af925794483afe9f26d1a0f4291 (127.25.109.3:41657)
I20260924 16:02:22.920444 859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.916831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:46785 (local address 127.25.109.2:35597)
0924 16:02:22.916978 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:22.916998 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:22.917124 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:22.917424 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:22.917439 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:22.917467 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:22.917616 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.917626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.918470 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.918474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.919332 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.919341 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.919987 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.919994 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.920095 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.920119 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 16:02:22.920195 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 16:02:22.920254 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 16:02:22.920485 1019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.916934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:35597 (local address 127.25.109.1:46785)
0924 16:02:22.917078 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:22.917083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:22.917100 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:02:22.917222 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:22.917227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:22.917277 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:22.917354 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:22.917737 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.918319 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.919465 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.919860 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.919954 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.920031 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:22.920233 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 16:02:22.920321 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 16:02:22.920552 1023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.917286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:48749 (local address 127.25.109.3:41657)
0924 16:02:22.917429 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:22.917434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:22.917447 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:02:22.917498 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:22.917500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:22.917543 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:22.917630 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:22.917953 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.918504 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.919517 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.919937 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.920056 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.920110 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:22.920323 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 16:02:22.920393 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 16:02:22.920575 861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.917168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:41657 (local address 127.25.109.2:48749)
0924 16:02:22.917287 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:22.917301 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:22.917360 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:22.917655 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:22.917657 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:22.917692 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:22.917817 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.917823 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.918628 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.918631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.919356 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:22.919364 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.920080 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.920084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:22.920161 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.920210 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 16:02:22.920287 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 16:02:22.920353 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 16:02:22.921310 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1456c04181bd4d2eba410d0138021a9e" candidate_uuid: "f758e6c10f3448cdb5d1e3ba31d30925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3fa4c2ab3d94a049605e22b6492d8ab" is_pre_election: true
I20260924 16:02:22.921657 671 raft_consensus.cc:2454] T 1456c04181bd4d2eba410d0138021a9e P c3fa4c2ab3d94a049605e22b6492d8ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f758e6c10f3448cdb5d1e3ba31d30925 in term 0.
I20260924 16:02:22.921656 945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1456c04181bd4d2eba410d0138021a9e" candidate_uuid: "f758e6c10f3448cdb5d1e3ba31d30925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c9c5af925794483afe9f26d1a0f4291" is_pre_election: true
I20260924 16:02:22.921963 945 raft_consensus.cc:2454] T 1456c04181bd4d2eba410d0138021a9e P 1c9c5af925794483afe9f26d1a0f4291 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f758e6c10f3448cdb5d1e3ba31d30925 in term 0.
I20260924 16:02:22.922187 742 leader_election.cc:302] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3fa4c2ab3d94a049605e22b6492d8ab, f758e6c10f3448cdb5d1e3ba31d30925; no voters:
I20260924 16:02:22.922518 1031 raft_consensus.cc:2781] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:02:22.922617 1031 raft_consensus.cc:482] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:02:22.922678 1031 raft_consensus.cc:3037] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:22.923815 1031 raft_consensus.cc:504] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3fa4c2ab3d94a049605e22b6492d8ab" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46785 } } peers { permanent_uuid: "f758e6c10f3448cdb5d1e3ba31d30925" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38957 } } peers { permanent_uuid: "1c9c5af925794483afe9f26d1a0f4291" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41657 } }
I20260924 16:02:22.924208 1031 leader_election.cc:288] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925 [CANDIDATE]: Term 1 election: Requested vote from peers c3fa4c2ab3d94a049605e22b6492d8ab (127.25.109.1:46785), 1c9c5af925794483afe9f26d1a0f4291 (127.25.109.3:41657)
I20260924 16:02:22.924542 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1456c04181bd4d2eba410d0138021a9e" candidate_uuid: "f758e6c10f3448cdb5d1e3ba31d30925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3fa4c2ab3d94a049605e22b6492d8ab"
I20260924 16:02:22.924582 945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1456c04181bd4d2eba410d0138021a9e" candidate_uuid: "f758e6c10f3448cdb5d1e3ba31d30925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c9c5af925794483afe9f26d1a0f4291"
I20260924 16:02:22.924692 671 raft_consensus.cc:3037] T 1456c04181bd4d2eba410d0138021a9e P c3fa4c2ab3d94a049605e22b6492d8ab [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:22.924691 945 raft_consensus.cc:3037] T 1456c04181bd4d2eba410d0138021a9e P 1c9c5af925794483afe9f26d1a0f4291 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:22.925997 945 raft_consensus.cc:2454] T 1456c04181bd4d2eba410d0138021a9e P 1c9c5af925794483afe9f26d1a0f4291 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f758e6c10f3448cdb5d1e3ba31d30925 in term 1.
I20260924 16:02:22.925997 671 raft_consensus.cc:2454] T 1456c04181bd4d2eba410d0138021a9e P c3fa4c2ab3d94a049605e22b6492d8ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f758e6c10f3448cdb5d1e3ba31d30925 in term 1.
I20260924 16:02:22.926287 742 leader_election.cc:302] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c9c5af925794483afe9f26d1a0f4291, f758e6c10f3448cdb5d1e3ba31d30925; no voters:
I20260924 16:02:22.926478 1031 raft_consensus.cc:2781] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:22.926741 1031 raft_consensus.cc:686] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925 [term 1 LEADER]: Becoming Leader. State: Replica: f758e6c10f3448cdb5d1e3ba31d30925, State: Running, Role: LEADER
I20260924 16:02:22.927325 1031 consensus_queue.cc:237] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3fa4c2ab3d94a049605e22b6492d8ab" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46785 } } peers { permanent_uuid: "f758e6c10f3448cdb5d1e3ba31d30925" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38957 } } peers { permanent_uuid: "1c9c5af925794483afe9f26d1a0f4291" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41657 } }
I20260924 16:02:22.930657 32220 catalog_manager.cc:5495] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925 reported cstate change: term changed from 0 to 1, leader changed from <none> to f758e6c10f3448cdb5d1e3ba31d30925 (127.25.109.2). New cstate: current_term: 1 leader_uuid: "f758e6c10f3448cdb5d1e3ba31d30925" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3fa4c2ab3d94a049605e22b6492d8ab" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f758e6c10f3448cdb5d1e3ba31d30925" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c9c5af925794483afe9f26d1a0f4291" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41657 } health_report { overall_health: UNKNOWN } } }
I20260924 16:02:22.963318 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.959470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40360 (local address 127.25.109.2:38957)
0924 16:02:22.959633 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:22.959638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:22.959649 (+ 11us) server_negotiation.cc:407] Connection header received
0924 16:02:22.959729 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:22.959731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:22.959785 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:22.959902 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:02:22.960190 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.960802 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.961711 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:22.961940 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:22.962060 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:22.962418 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:02:22.962879 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:02:22.963014 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:22.963095 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 16:02:22.963158 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 16:02:22.969732 945 raft_consensus.cc:1261] T 1456c04181bd4d2eba410d0138021a9e P 1c9c5af925794483afe9f26d1a0f4291 [term 1 FOLLOWER]: Refusing update from remote peer f758e6c10f3448cdb5d1e3ba31d30925: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:02:22.969741 671 raft_consensus.cc:1261] T 1456c04181bd4d2eba410d0138021a9e P c3fa4c2ab3d94a049605e22b6492d8ab [term 1 FOLLOWER]: Refusing update from remote peer f758e6c10f3448cdb5d1e3ba31d30925: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:02:22.970530 1031 consensus_queue.cc:1035] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3fa4c2ab3d94a049605e22b6492d8ab" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:22.970777 1033 consensus_queue.cc:1035] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c9c5af925794483afe9f26d1a0f4291" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:22.980338 1036 mvcc.cc:205] Tried to move back new op lower bound from 7332928483195801600 to 7332928483032911872. Current Snapshot: MvccSnapshot[applied={T|T < 7332928483195801600}]
I20260924 16:02:22.984993 1037 mvcc.cc:205] Tried to move back new op lower bound from 7332928483195801600 to 7332928483032911872. Current Snapshot: MvccSnapshot[applied={T|T < 7332928483195801600}]
I20260924 16:02:22.991175 32220 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:02:22.991365 32220 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:02:22.996567 32220 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bda7fa2c9ca5483ea36450964edfe158: Sending DeleteTablet for 3 replicas of tablet 1456c04181bd4d2eba410d0138021a9e
I20260924 16:02:22.998045 925 tablet_service.cc:1514] Processing DeleteTablet for tablet 1456c04181bd4d2eba410d0138021a9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:22 UTC) from {username='oryx', principal='oryx/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:50188
I20260924 16:02:22.998139 789 tablet_service.cc:1514] Processing DeleteTablet for tablet 1456c04181bd4d2eba410d0138021a9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:22 UTC) from {username='oryx', principal='oryx/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:40346
I20260924 16:02:22.998189 651 tablet_service.cc:1514] Processing DeleteTablet for tablet 1456c04181bd4d2eba410d0138021a9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:22 UTC) from {username='oryx', principal='oryx/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:40618
I20260924 16:02:22.998698 1046 tablet_replica.cc:330] T 1456c04181bd4d2eba410d0138021a9e P c3fa4c2ab3d94a049605e22b6492d8ab: stopping tablet replica
I20260924 16:02:22.999035 1046 raft_consensus.cc:2229] T 1456c04181bd4d2eba410d0138021a9e P c3fa4c2ab3d94a049605e22b6492d8ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:22.999287 1045 tablet_replica.cc:330] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925: stopping tablet replica
I20260924 16:02:22.999435 1046 raft_consensus.cc:2258] T 1456c04181bd4d2eba410d0138021a9e P c3fa4c2ab3d94a049605e22b6492d8ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:22.999944 1045 raft_consensus.cc:2229] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:02:23.000839 1045 raft_consensus.cc:2258] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:23.001001 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 32189
I20260924 16:02:23.002241 1046 ts_tablet_manager.cc:1905] T 1456c04181bd4d2eba410d0138021a9e P c3fa4c2ab3d94a049605e22b6492d8ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:23.003336 1045 ts_tablet_manager.cc:1905] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:23.005944 1046 ts_tablet_manager.cc:1918] T 1456c04181bd4d2eba410d0138021a9e P c3fa4c2ab3d94a049605e22b6492d8ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:02:23.006043 1046 log.cc:1196] T 1456c04181bd4d2eba410d0138021a9e P c3fa4c2ab3d94a049605e22b6492d8ab: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-0/wal/wals/1456c04181bd4d2eba410d0138021a9e
I20260924 16:02:23.006108 1045 ts_tablet_manager.cc:1918] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:02:23.006199 1045 log.cc:1196] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/wal/wals/1456c04181bd4d2eba410d0138021a9e
I20260924 16:02:23.006371 1046 ts_tablet_manager.cc:1939] T 1456c04181bd4d2eba410d0138021a9e P c3fa4c2ab3d94a049605e22b6492d8ab: Deleting consensus metadata
I20260924 16:02:23.006546 1045 ts_tablet_manager.cc:1939] T 1456c04181bd4d2eba410d0138021a9e P f758e6c10f3448cdb5d1e3ba31d30925: Deleting consensus metadata
I20260924 16:02:23.007889 1047 tablet_replica.cc:330] T 1456c04181bd4d2eba410d0138021a9e P 1c9c5af925794483afe9f26d1a0f4291: stopping tablet replica
I20260924 16:02:23.008284 1047 raft_consensus.cc:2229] T 1456c04181bd4d2eba410d0138021a9e P 1c9c5af925794483afe9f26d1a0f4291 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:23.008693 1047 raft_consensus.cc:2258] T 1456c04181bd4d2eba410d0138021a9e P 1c9c5af925794483afe9f26d1a0f4291 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 16:02:23.010388 745 connection.cc:535] server connection from 127.0.0.1:40346 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 16:02:23.010792 604 connection.cc:535] server connection from 127.0.0.1:40618 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 16:02:23.010969 1047 ts_tablet_manager.cc:1905] T 1456c04181bd4d2eba410d0138021a9e P 1c9c5af925794483afe9f26d1a0f4291: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:23.014010 1047 ts_tablet_manager.cc:1918] T 1456c04181bd4d2eba410d0138021a9e P 1c9c5af925794483afe9f26d1a0f4291: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:02:23.014117 1047 log.cc:1196] T 1456c04181bd4d2eba410d0138021a9e P 1c9c5af925794483afe9f26d1a0f4291: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/wal/wals/1456c04181bd4d2eba410d0138021a9e
I20260924 16:02:23.014470 1047 ts_tablet_manager.cc:1939] T 1456c04181bd4d2eba410d0138021a9e P 1c9c5af925794483afe9f26d1a0f4291: Deleting consensus metadata
W20260924 16:02:23.015733 881 connection.cc:441] server connection from 127.0.0.1:50188 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50188 (request call id 1) could send its response
I20260924 16:02:23.020366 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:22.999555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40440 (local address 127.25.109.61:43277)
0924 16:02:22.999676 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:22.999681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:23.009288 (+ 9607us) server_negotiation.cc:407] Connection header received
0924 16:02:23.009430 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:23.009434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:23.009493 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:23.009584 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:23.010679 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:23.011539 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:23.013298 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:23.013667 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:23.013745 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:23.016274 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:23.016278 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:23.016281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:23.016311 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:23.018433 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:23.018933 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:23.018937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:23.018939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:23.019038 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:23.019328 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:23.019332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:23.019334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:23.019541 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:23.019642 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:23.020062 (+ 420us) server_negotiation.cc:299] Negotiation successful
0924 16:02:23.020194 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 16:02:23.025918 1004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:23.014828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49398 (local address 127.25.109.60:44917)
0924 16:02:23.014966 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:23.014971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:23.016297 (+ 1326us) server_negotiation.cc:407] Connection header received
0924 16:02:23.016549 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:23.016553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:23.016622 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:23.016682 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:23.018038 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:23.018797 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:23.019881 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:23.020214 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:23.020289 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:23.021929 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:23.021933 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:23.021935 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:23.021963 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:23.023903 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:23.024465 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:23.024468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:23.024469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:23.024553 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:23.024925 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:23.024928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:23.024930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:23.025123 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:23.025240 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:23.025642 (+ 402us) server_negotiation.cc:299] Negotiation successful
0924 16:02:23.025768 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 16:02:23.025918 478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:23.015621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40450 (local address 127.25.109.61:43277)
0924 16:02:23.015733 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:23.015737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:23.016778 (+ 1041us) server_negotiation.cc:407] Connection header received
0924 16:02:23.016821 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:23.016824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:23.016869 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:23.016949 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:23.017648 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:23.018456 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:23.019645 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:23.019972 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:23.020038 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:23.021890 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:23.021894 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:23.021897 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:23.021922 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:23.023820 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:23.024461 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:23.024465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:23.024467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:23.024554 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:23.024909 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:23.024912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:23.024914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:23.025134 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:23.025239 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:23.025639 (+ 400us) server_negotiation.cc:299] Negotiation successful
0924 16:02:23.025758 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 16:02:23.032832 724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:23.000183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49392 (local address 127.25.109.60:44917)
0924 16:02:23.000298 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:23.000303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:23.023050 (+ 22747us) server_negotiation.cc:407] Connection header received
0924 16:02:23.023216 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:23.023219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:23.023262 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:23.023311 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:23.024078 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:23.024862 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:23.026703 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:23.027056 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:23.027129 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:23.028951 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:23.028954 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:23.028956 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:23.028977 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:23.030521 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:23.031699 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:23.031703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:23.031705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:23.031793 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:23.032084 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:23.032087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:23.032088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:23.032264 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:23.032362 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:23.032623 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 16:02:23.032717 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 16:02:23.984881 1056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:23.984283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.25.109.3:58581)
0924 16:02:23.984631 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:23.984735 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":116,"threads_started":1}
W20260924 16:02:23.985414 992 heartbeater.cc:644] Failed to heartbeat to 127.25.109.62:33903 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
I20260924 16:02:23.989425 1057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:23.988941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.25.109.2:39321)
0924 16:02:23.989210 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:23.989313 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":79,"threads_started":1}
W20260924 16:02:23.989718 856 heartbeater.cc:644] Failed to heartbeat to 127.25.109.62:33903 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
I20260924 16:02:23.990597 1058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:23.990214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.25.109.1:36781)
0924 16:02:23.990421 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:23.990517 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":62,"threads_started":1}
W20260924 16:02:23.990950 719 heartbeater.cc:644] Failed to heartbeat to 127.25.109.62:33903 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
I20260924 16:02:24.496297 1059 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bda7fa2c9ca5483ea36450964edfe158)
I20260924 16:02:24.496587 1059 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: false } }
I20260924 16:02:24.497364 1060 raft_consensus.cc:482] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bda7fa2c9ca5483ea36450964edfe158)
I20260924 16:02:24.497560 1060 raft_consensus.cc:504] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: false } }
I20260924 16:02:24.497709 1059 leader_election.cc:288] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bda7fa2c9ca5483ea36450964edfe158 (127.25.109.62:33903), daf9b928e16b41c787926bfe44bf616e (127.25.109.61:43277)
I20260924 16:02:24.498392 1062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:24.497884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.0.0.1:54888)
0924 16:02:24.498204 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:24.498287 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":81,"threads_started":1}
W20260924 16:02:24.498610 522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
I20260924 16:02:24.498692 1060 leader_election.cc:288] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bda7fa2c9ca5483ea36450964edfe158 (127.25.109.62:33903), 9450614f7a7c4bcebf504f2db67ba64e (127.25.109.60:44917)
I20260924 16:02:24.499682 1062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:24.499426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.0.0.1:54900)
0924 16:02:24.499538 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:24.499593 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 16:02:24.499804 522 leader_election.cc:334] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bda7fa2c9ca5483ea36450964edfe158 (127.25.109.62:33903): Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
I20260924 16:02:24.499957 1067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:24.499067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.0.0.1:54896)
0924 16:02:24.499554 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:24.499642 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":63,"threads_started":1}
W20260924 16:02:24.500139 340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
I20260924 16:02:24.501025 1067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:24.500795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.0.0.1:54910)
0924 16:02:24.500933 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:24.500972 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260924 16:02:24.501144 340 leader_election.cc:334] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bda7fa2c9ca5483ea36450964edfe158 (127.25.109.62:33903): Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
I20260924 16:02:24.502777 1063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:24.498028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56108 (local address 127.25.109.61:43277)
0924 16:02:24.498297 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:24.498303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:24.498415 (+ 112us) server_negotiation.cc:407] Connection header received
0924 16:02:24.498670 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:24.498673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:24.498759 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:24.498864 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:24.499289 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:24.500226 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:24.501702 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:24.502199 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:24.502281 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:24.502332 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:24.502570 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 16:02:24.502662 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":194,"thread_start_us":66,"threads_started":1}
I20260924 16:02:24.502815 1061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:24.497931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:43277 (local address 127.0.0.1:56108)
0924 16:02:24.498310 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:24.498325 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:24.498529 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:24.499007 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:24.499010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:24.499028 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:24.499156 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:24.499164 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:24.500331 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:24.500334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:24.501568 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:24.501577 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:24.502314 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:24.502318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:24.502414 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:24.502458 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 16:02:24.502572 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 16:02:24.502708 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":55,"thread_start_us":66,"threads_started":1}
I20260924 16:02:24.503407 366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9450614f7a7c4bcebf504f2db67ba64e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "daf9b928e16b41c787926bfe44bf616e" is_pre_election: true
I20260924 16:02:24.503672 366 raft_consensus.cc:2454] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9450614f7a7c4bcebf504f2db67ba64e in term 1.
I20260924 16:02:24.503814 1065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:24.498865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:44917 (local address 127.0.0.1:56316)
0924 16:02:24.499238 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:24.499253 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:24.499350 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:24.499792 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:24.499798 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:24.499816 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:24.500015 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:24.500026 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:24.501291 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:24.501294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:24.502371 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:24.502381 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:24.503305 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:24.503309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:24.503401 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:24.503479 (+ 78us) client_negotiation.cc:769] Sending connection context
0924 16:02:24.503572 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 16:02:24.503649 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":41,"thread_start_us":66,"threads_started":1}
I20260924 16:02:24.503825 1066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:24.499003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56316 (local address 127.25.109.60:44917)
0924 16:02:24.499482 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:24.499492 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:24.499505 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:02:24.499555 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:24.499557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:24.499600 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:24.499677 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:24.500254 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:24.501161 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:24.502493 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:24.503169 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:24.503278 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:24.503322 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:24.503610 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 16:02:24.503694 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":80,"threads_started":1}
I20260924 16:02:24.504169 523 leader_election.cc:302] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9450614f7a7c4bcebf504f2db67ba64e, daf9b928e16b41c787926bfe44bf616e; no voters: bda7fa2c9ca5483ea36450964edfe158
I20260924 16:02:24.504211 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "daf9b928e16b41c787926bfe44bf616e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9450614f7a7c4bcebf504f2db67ba64e" is_pre_election: true
I20260924 16:02:24.504407 547 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate daf9b928e16b41c787926bfe44bf616e in term 1.
I20260924 16:02:24.504561 1059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 16:02:24.504624 1059 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [term 1 FOLLOWER]: Starting leader election (detected failure of leader bda7fa2c9ca5483ea36450964edfe158)
I20260924 16:02:24.504688 1059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [term 1 FOLLOWER]: Advancing to term 2
I20260924 16:02:24.504813 342 leader_election.cc:302] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9450614f7a7c4bcebf504f2db67ba64e, daf9b928e16b41c787926bfe44bf616e; no voters: bda7fa2c9ca5483ea36450964edfe158
I20260924 16:02:24.505069 1060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 16:02:24.505138 1060 raft_consensus.cc:482] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [term 1 FOLLOWER]: Starting leader election (detected failure of leader bda7fa2c9ca5483ea36450964edfe158)
I20260924 16:02:24.505178 1060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [term 1 FOLLOWER]: Advancing to term 2
I20260924 16:02:24.505903 1059 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: false } }
I20260924 16:02:24.506282 1059 leader_election.cc:288] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [CANDIDATE]: Term 2 election: Requested vote from peers bda7fa2c9ca5483ea36450964edfe158 (127.25.109.62:33903), daf9b928e16b41c787926bfe44bf616e (127.25.109.61:43277)
I20260924 16:02:24.506199 1060 raft_consensus.cc:504] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: false } }
I20260924 16:02:24.506606 366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9450614f7a7c4bcebf504f2db67ba64e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "daf9b928e16b41c787926bfe44bf616e"
I20260924 16:02:24.506736 1061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:24.506405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.0.0.1:54924)
0924 16:02:24.506499 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:24.506669 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 16:02:24.506793 366 raft_consensus.cc:2379] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9450614f7a7c4bcebf504f2db67ba64e in current term 2: Already voted for candidate daf9b928e16b41c787926bfe44bf616e in this term.
I20260924 16:02:24.506960 1065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:24.506710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.0.0.1:54930)
0924 16:02:24.506852 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:24.506900 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 16:02:24.506930 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "daf9b928e16b41c787926bfe44bf616e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9450614f7a7c4bcebf504f2db67ba64e"
I20260924 16:02:24.507107 547 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate daf9b928e16b41c787926bfe44bf616e in current term 2: Already voted for candidate 9450614f7a7c4bcebf504f2db67ba64e in this term.
I20260924 16:02:24.506628 1060 leader_election.cc:288] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [CANDIDATE]: Term 2 election: Requested vote from peers bda7fa2c9ca5483ea36450964edfe158 (127.25.109.62:33903), 9450614f7a7c4bcebf504f2db67ba64e (127.25.109.60:44917)
I20260924 16:02:24.507606 1061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:24.507380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.0.0.1:54936)
0924 16:02:24.507471 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:24.507539 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 16:02:24.507706 522 leader_election.cc:334] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bda7fa2c9ca5483ea36450964edfe158 (127.25.109.62:33903): Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
I20260924 16:02:24.507817 522 leader_election.cc:302] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9450614f7a7c4bcebf504f2db67ba64e; no voters: bda7fa2c9ca5483ea36450964edfe158, daf9b928e16b41c787926bfe44bf616e
I20260924 16:02:24.507990 1059 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 16:02:24.508239 1065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:24.507614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.0.0.1:54938)
0924 16:02:24.508136 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:24.508192 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446}
W20260924 16:02:24.508384 340 leader_election.cc:334] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bda7fa2c9ca5483ea36450964edfe158 (127.25.109.62:33903): Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
I20260924 16:02:24.508498 340 leader_election.cc:302] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: daf9b928e16b41c787926bfe44bf616e; no voters: 9450614f7a7c4bcebf504f2db67ba64e, bda7fa2c9ca5483ea36450964edfe158
I20260924 16:02:24.508610 1060 raft_consensus.cc:2726] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 16:02:24.986677 1069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:24.986103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.25.109.3:50387)
0924 16:02:24.986453 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:24.986536 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":129,"threads_started":1}
I20260924 16:02:24.990942 1070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:24.990320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.25.109.2:33171)
0924 16:02:24.990735 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:24.990812 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":104,"threads_started":1}
I20260924 16:02:24.991870 1071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:24.991508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.25.109.1:41025)
0924 16:02:24.991745 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:24.991796 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":67,"threads_started":1}
I20260924 16:02:25.988221 1072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:25.987697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.25.109.3:34861)
0924 16:02:25.988015 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:25.988098 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":93,"threads_started":1}
I20260924 16:02:25.992161 1073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:25.991672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.25.109.2:56549)
0924 16:02:25.991947 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:25.992019 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":82,"threads_started":1}
I20260924 16:02:25.992933 1074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:25.992499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.25.109.1:43991)
0924 16:02:25.992809 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:25.992857 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":118,"threads_started":1}
I20260924 16:02:26.623561 1075 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:02:26.623759 1075 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: false } }
I20260924 16:02:26.624244 1075 leader_election.cc:288] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bda7fa2c9ca5483ea36450964edfe158 (127.25.109.62:33903), daf9b928e16b41c787926bfe44bf616e (127.25.109.61:43277)
I20260924 16:02:26.624752 366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9450614f7a7c4bcebf504f2db67ba64e" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "daf9b928e16b41c787926bfe44bf616e" is_pre_election: true
I20260924 16:02:26.624930 366 raft_consensus.cc:2454] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9450614f7a7c4bcebf504f2db67ba64e in term 2.
I20260924 16:02:26.624948 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.624517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.0.0.1:55190)
0924 16:02:26.624762 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:26.624826 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":62,"threads_started":1}
I20260924 16:02:26.625310 523 leader_election.cc:302] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9450614f7a7c4bcebf504f2db67ba64e, daf9b928e16b41c787926bfe44bf616e; no voters:
I20260924 16:02:26.625742 1075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 16:02:26.625811 1075 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:02:26.625845 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.625601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.0.0.1:55200)
0924 16:02:26.625711 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:26.625772 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 16:02:26.625852 1075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [term 2 FOLLOWER]: Advancing to term 3
W20260924 16:02:26.626019 522 leader_election.cc:334] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bda7fa2c9ca5483ea36450964edfe158 (127.25.109.62:33903): Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
I20260924 16:02:26.627107 1075 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: false } }
I20260924 16:02:26.627408 1075 leader_election.cc:288] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [CANDIDATE]: Term 3 election: Requested vote from peers bda7fa2c9ca5483ea36450964edfe158 (127.25.109.62:33903), daf9b928e16b41c787926bfe44bf616e (127.25.109.61:43277)
I20260924 16:02:26.627820 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.627568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.0.0.1:55206)
0924 16:02:26.627700 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:26.627756 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 16:02:26.627830 366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9450614f7a7c4bcebf504f2db67ba64e" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "daf9b928e16b41c787926bfe44bf616e"
I20260924 16:02:26.627992 366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [term 2 FOLLOWER]: Advancing to term 3
I20260924 16:02:26.628487 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.628260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.0.0.1:55218)
0924 16:02:26.628360 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:26.628426 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 16:02:26.628587 522 leader_election.cc:334] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer bda7fa2c9ca5483ea36450964edfe158 (127.25.109.62:33903): Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
I20260924 16:02:26.629170 366 raft_consensus.cc:2454] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9450614f7a7c4bcebf504f2db67ba64e in term 3.
I20260924 16:02:26.629482 523 leader_election.cc:302] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9450614f7a7c4bcebf504f2db67ba64e, daf9b928e16b41c787926bfe44bf616e; no voters: bda7fa2c9ca5483ea36450964edfe158
I20260924 16:02:26.629650 1075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [term 3 FOLLOWER]: Leader election won for term 3
I20260924 16:02:26.629834 1075 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [term 3 LEADER]: Becoming Leader. State: Replica: 9450614f7a7c4bcebf504f2db67ba64e, State: Running, Role: LEADER
I20260924 16:02:26.630112 1075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: false } }
I20260924 16:02:26.631356 1078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9450614f7a7c4bcebf504f2db67ba64e. Latest consensus state: current_term: 3 leader_uuid: "9450614f7a7c4bcebf504f2db67ba64e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: false } } }
I20260924 16:02:26.631493 1078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:26.631794 1080 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:02:26.634137 1080 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:02:26.634547 1080 catalog_manager.cc:1259] Loaded cluster ID: 3cb794533a7b41a6955bf42add1fa079
I20260924 16:02:26.634603 1080 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:02:26.635255 1080 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:02:26.635659 1080 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e: Loaded TSK: 0
I20260924 16:02:26.636521 1080 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:02:26.659459 538 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:02:26.663357 538 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:02:26.666500 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.666257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.0.0.1:55232)
0924 16:02:26.666373 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:26.666428 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 16:02:26.666648 366 raft_consensus.cc:1261] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [term 3 FOLLOWER]: Refusing update from remote peer 9450614f7a7c4bcebf504f2db67ba64e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260924 16:02:26.667279 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.666973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.0.0.1:55242)
0924 16:02:26.667108 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:26.667180 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 16:02:26.667177 1075 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [LEADER]: Connected to new peer: Peer: permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260924 16:02:26.667496 522 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e -> Peer bda7fa2c9ca5483ea36450964edfe158 (127.25.109.62:33903): Couldn't send request to peer bda7fa2c9ca5483ea36450964edfe158. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 16:02:26.668735 1082 sys_catalog.cc:455] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9450614f7a7c4bcebf504f2db67ba64e. Latest consensus state: current_term: 3 leader_uuid: "9450614f7a7c4bcebf504f2db67ba64e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: false } } }
I20260924 16:02:26.668879 1082 sys_catalog.cc:458] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:02:26.670081 1075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "9450614f7a7c4bcebf504f2db67ba64e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: false } } }
I20260924 16:02:26.670202 1075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:26.670524 1082 sys_catalog.cc:455] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "9450614f7a7c4bcebf504f2db67ba64e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bda7fa2c9ca5483ea36450964edfe158" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33903 } } peers { permanent_uuid: "daf9b928e16b41c787926bfe44bf616e" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 43277 } attrs { promote: false } } peers { permanent_uuid: "9450614f7a7c4bcebf504f2db67ba64e" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 44917 } attrs { promote: false } } }
I20260924 16:02:26.670711 1082 sys_catalog.cc:458] T 00000000000000000000000000000000 P daf9b928e16b41c787926bfe44bf616e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:02:26.680997 1088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.676920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38336 (local address 127.25.109.3:41657)
0924 16:02:26.677198 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:26.677203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:26.677217 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:02:26.677271 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:26.677275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:26.677326 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:26.677460 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:26.677745 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.678547 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.679933 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.680358 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.680511 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:26.680587 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:26.680790 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 16:02:26.680867 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":91,"threads_started":1}
I20260924 16:02:26.680986 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.676649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:41657 (local address 127.0.0.1:38336)
0924 16:02:26.676767 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:26.676785 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:26.676887 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:26.677475 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:26.677478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:26.677494 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:26.677604 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:26.677611 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.678679 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.678682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:26.679800 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:26.679810 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.680518 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.680522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:26.680635 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:26.680658 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 16:02:26.680753 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 16:02:26.680831 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":4}
I20260924 16:02:26.681962 1091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.677553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:46785 (local address 127.0.0.1:56150)
0924 16:02:26.678222 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:26.678235 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:26.678324 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:26.678576 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:26.678579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:26.678594 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:26.678666 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:26.678670 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.679422 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.679429 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:26.680523 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:26.680546 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.681583 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.681597 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:26.681685 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:26.681712 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 16:02:26.681790 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 16:02:26.681857 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":93,"threads_started":1}
I20260924 16:02:26.682005 925 tablet_service.cc:1467] Processing CreateTablet for tablet 8c4c81cddfb141ab8b69c5f778278710 (DEFAULT_TABLE table=test-table [id=a7745e2bb4bf48ccb9478d1ffdc32794]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:26.682345 925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c4c81cddfb141ab8b69c5f778278710. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:26.682374 1092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.677907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56150 (local address 127.25.109.1:46785)
0924 16:02:26.678169 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:26.678174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:26.678311 (+ 137us) server_negotiation.cc:407] Connection header received
0924 16:02:26.678406 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:26.678409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:26.678454 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:26.678541 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:26.678777 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.679290 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.680677 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.681457 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.681563 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:26.681651 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:26.682152 (+ 501us) server_negotiation.cc:299] Negotiation successful
0924 16:02:26.682239 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":77,"threads_started":1}
I20260924 16:02:26.682734 1087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.676907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:38957 (local address 127.0.0.1:43148)
0924 16:02:26.677701 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:26.677716 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:26.677797 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:26.678178 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:26.678182 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:26.678198 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:26.678311 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:26.678317 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.679567 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.679570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:26.681543 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:26.681552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.682370 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.682375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:26.682469 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:26.682500 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 16:02:26.682576 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 16:02:26.682632 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":115,"threads_started":1}
I20260924 16:02:26.682860 1089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.677258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43148 (local address 127.25.109.2:38957)
0924 16:02:26.677776 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:26.677781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:26.677799 (+ 18us) server_negotiation.cc:407] Connection header received
0924 16:02:26.677905 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:26.677908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:26.677958 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:26.678053 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:26.678692 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.679463 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.681694 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.682243 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.682369 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:26.682438 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:26.682623 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 16:02:26.682721 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":96,"threads_started":1}
I20260924 16:02:26.683445 789 tablet_service.cc:1467] Processing CreateTablet for tablet 8c4c81cddfb141ab8b69c5f778278710 (DEFAULT_TABLE table=test-table [id=a7745e2bb4bf48ccb9478d1ffdc32794]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:26.683738 789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c4c81cddfb141ab8b69c5f778278710. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:26.686143 651 tablet_service.cc:1467] Processing CreateTablet for tablet 8c4c81cddfb141ab8b69c5f778278710 (DEFAULT_TABLE table=test-table [id=a7745e2bb4bf48ccb9478d1ffdc32794]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:26.686305 1093 tablet_bootstrap.cc:492] T 8c4c81cddfb141ab8b69c5f778278710 P 1c9c5af925794483afe9f26d1a0f4291: Bootstrap starting.
I20260924 16:02:26.686524 651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c4c81cddfb141ab8b69c5f778278710. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:26.686712 1094 tablet_bootstrap.cc:492] T 8c4c81cddfb141ab8b69c5f778278710 P f758e6c10f3448cdb5d1e3ba31d30925: Bootstrap starting.
I20260924 16:02:26.687456 1093 tablet_bootstrap.cc:654] T 8c4c81cddfb141ab8b69c5f778278710 P 1c9c5af925794483afe9f26d1a0f4291: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:26.687602 1094 tablet_bootstrap.cc:654] T 8c4c81cddfb141ab8b69c5f778278710 P f758e6c10f3448cdb5d1e3ba31d30925: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:26.688786 1093 tablet_bootstrap.cc:492] T 8c4c81cddfb141ab8b69c5f778278710 P 1c9c5af925794483afe9f26d1a0f4291: No bootstrap required, opened a new log
I20260924 16:02:26.688803 1094 tablet_bootstrap.cc:492] T 8c4c81cddfb141ab8b69c5f778278710 P f758e6c10f3448cdb5d1e3ba31d30925: No bootstrap required, opened a new log
I20260924 16:02:26.688827 1097 tablet_bootstrap.cc:492] T 8c4c81cddfb141ab8b69c5f778278710 P c3fa4c2ab3d94a049605e22b6492d8ab: Bootstrap starting.
I20260924 16:02:26.688885 1093 ts_tablet_manager.cc:1397] T 8c4c81cddfb141ab8b69c5f778278710 P 1c9c5af925794483afe9f26d1a0f4291: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 16:02:26.688889 1094 ts_tablet_manager.cc:1397] T 8c4c81cddfb141ab8b69c5f778278710 P f758e6c10f3448cdb5d1e3ba31d30925: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 16:02:26.689283 1094 raft_consensus.cc:348] T 8c4c81cddfb141ab8b69c5f778278710 P f758e6c10f3448cdb5d1e3ba31d30925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c9c5af925794483afe9f26d1a0f4291" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41657 } } peers { permanent_uuid: "f758e6c10f3448cdb5d1e3ba31d30925" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38957 } } peers { permanent_uuid: "c3fa4c2ab3d94a049605e22b6492d8ab" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46785 } }
I20260924 16:02:26.689417 1094 raft_consensus.cc:374] T 8c4c81cddfb141ab8b69c5f778278710 P f758e6c10f3448cdb5d1e3ba31d30925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:26.689354 1093 raft_consensus.cc:348] T 8c4c81cddfb141ab8b69c5f778278710 P 1c9c5af925794483afe9f26d1a0f4291 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c9c5af925794483afe9f26d1a0f4291" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41657 } } peers { permanent_uuid: "f758e6c10f3448cdb5d1e3ba31d30925" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38957 } } peers { permanent_uuid: "c3fa4c2ab3d94a049605e22b6492d8ab" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46785 } }
I20260924 16:02:26.689461 1094 raft_consensus.cc:729] T 8c4c81cddfb141ab8b69c5f778278710 P f758e6c10f3448cdb5d1e3ba31d30925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f758e6c10f3448cdb5d1e3ba31d30925, State: Initialized, Role: FOLLOWER
I20260924 16:02:26.689467 1093 raft_consensus.cc:374] T 8c4c81cddfb141ab8b69c5f778278710 P 1c9c5af925794483afe9f26d1a0f4291 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:26.689499 1093 raft_consensus.cc:729] T 8c4c81cddfb141ab8b69c5f778278710 P 1c9c5af925794483afe9f26d1a0f4291 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c9c5af925794483afe9f26d1a0f4291, State: Initialized, Role: FOLLOWER
I20260924 16:02:26.689587 1094 consensus_queue.cc:260] T 8c4c81cddfb141ab8b69c5f778278710 P f758e6c10f3448cdb5d1e3ba31d30925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c9c5af925794483afe9f26d1a0f4291" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41657 } } peers { permanent_uuid: "f758e6c10f3448cdb5d1e3ba31d30925" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38957 } } peers { permanent_uuid: "c3fa4c2ab3d94a049605e22b6492d8ab" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46785 } }
I20260924 16:02:26.689616 1093 consensus_queue.cc:260] T 8c4c81cddfb141ab8b69c5f778278710 P 1c9c5af925794483afe9f26d1a0f4291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c9c5af925794483afe9f26d1a0f4291" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41657 } } peers { permanent_uuid: "f758e6c10f3448cdb5d1e3ba31d30925" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38957 } } peers { permanent_uuid: "c3fa4c2ab3d94a049605e22b6492d8ab" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46785 } }
I20260924 16:02:26.689975 1093 ts_tablet_manager.cc:1428] T 8c4c81cddfb141ab8b69c5f778278710 P 1c9c5af925794483afe9f26d1a0f4291: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:02:26.689985 1094 ts_tablet_manager.cc:1428] T 8c4c81cddfb141ab8b69c5f778278710 P f758e6c10f3448cdb5d1e3ba31d30925: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 16:02:26.690533 1097 tablet_bootstrap.cc:654] T 8c4c81cddfb141ab8b69c5f778278710 P c3fa4c2ab3d94a049605e22b6492d8ab: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:26.691901 1097 tablet_bootstrap.cc:492] T 8c4c81cddfb141ab8b69c5f778278710 P c3fa4c2ab3d94a049605e22b6492d8ab: No bootstrap required, opened a new log
I20260924 16:02:26.691996 1097 ts_tablet_manager.cc:1397] T 8c4c81cddfb141ab8b69c5f778278710 P c3fa4c2ab3d94a049605e22b6492d8ab: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 16:02:26.692550 1097 raft_consensus.cc:348] T 8c4c81cddfb141ab8b69c5f778278710 P c3fa4c2ab3d94a049605e22b6492d8ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c9c5af925794483afe9f26d1a0f4291" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41657 } } peers { permanent_uuid: "f758e6c10f3448cdb5d1e3ba31d30925" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38957 } } peers { permanent_uuid: "c3fa4c2ab3d94a049605e22b6492d8ab" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46785 } }
I20260924 16:02:26.692690 1097 raft_consensus.cc:374] T 8c4c81cddfb141ab8b69c5f778278710 P c3fa4c2ab3d94a049605e22b6492d8ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:26.692726 1097 raft_consensus.cc:729] T 8c4c81cddfb141ab8b69c5f778278710 P c3fa4c2ab3d94a049605e22b6492d8ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3fa4c2ab3d94a049605e22b6492d8ab, State: Initialized, Role: FOLLOWER
I20260924 16:02:26.692857 1097 consensus_queue.cc:260] T 8c4c81cddfb141ab8b69c5f778278710 P c3fa4c2ab3d94a049605e22b6492d8ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c9c5af925794483afe9f26d1a0f4291" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41657 } } peers { permanent_uuid: "f758e6c10f3448cdb5d1e3ba31d30925" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38957 } } peers { permanent_uuid: "c3fa4c2ab3d94a049605e22b6492d8ab" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46785 } }
I20260924 16:02:26.692914 1102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.691947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.25.109.2:40027)
0924 16:02:26.692739 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:26.692811 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":90,"threads_started":1}
I20260924 16:02:26.693199 1097 ts_tablet_manager.cc:1428] T 8c4c81cddfb141ab8b69c5f778278710 P c3fa4c2ab3d94a049605e22b6492d8ab: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 16:02:26.693760 855 heartbeater.cc:503] Master 127.25.109.60:44917 was elected leader, sending a full tablet report...
I20260924 16:02:26.694767 991 heartbeater.cc:503] Master 127.25.109.60:44917 was elected leader, sending a full tablet report...
I20260924 16:02:26.694998 1104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.694226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.25.109.1:42311)
0924 16:02:26.694852 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:26.694910 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":90,"threads_started":1}
I20260924 16:02:26.695900 718 heartbeater.cc:503] Master 127.25.109.60:44917 was elected leader, sending a full tablet report...
I20260924 16:02:26.695983 1104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.695615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.25.109.1:34713)
0924 16:02:26.695844 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:26.695891 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260924 16:02:26.696851 1101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.691165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.25.109.3:56445)
0924 16:02:26.696651 (+ 5486us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:26.696719 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2224,"thread_start_us":91,"threads_started":1}
I20260924 16:02:26.697095 1102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.696744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.25.109.2:54663)
0924 16:02:26.696877 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:26.697024 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 16:02:26.697885 1101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.697650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.25.109.3:52429)
0924 16:02:26.697792 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:26.697833 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260924 16:02:26.761469 994 tablet.cc:2369] T 8c4c81cddfb141ab8b69c5f778278710 P 1c9c5af925794483afe9f26d1a0f4291: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:02:26.770979 1103 raft_consensus.cc:482] T 8c4c81cddfb141ab8b69c5f778278710 P c3fa4c2ab3d94a049605e22b6492d8ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:02:26.771139 1103 raft_consensus.cc:504] T 8c4c81cddfb141ab8b69c5f778278710 P c3fa4c2ab3d94a049605e22b6492d8ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c9c5af925794483afe9f26d1a0f4291" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41657 } } peers { permanent_uuid: "f758e6c10f3448cdb5d1e3ba31d30925" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38957 } } peers { permanent_uuid: "c3fa4c2ab3d94a049605e22b6492d8ab" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46785 } }
I20260924 16:02:26.772341 1103 leader_election.cc:288] T 8c4c81cddfb141ab8b69c5f778278710 P c3fa4c2ab3d94a049605e22b6492d8ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c9c5af925794483afe9f26d1a0f4291 (127.25.109.3:41657), f758e6c10f3448cdb5d1e3ba31d30925 (127.25.109.2:38957)
I20260924 16:02:26.776391 1089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.772917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:45541 (local address 127.25.109.2:38957)
0924 16:02:26.773066 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:26.773073 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:26.773089 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:02:26.773149 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:26.773152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:26.773204 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:26.773288 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:26.773642 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.774185 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.775355 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.775766 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.775880 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:26.775945 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:26.776187 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 16:02:26.776268 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:02:26.776391 1104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.772622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:38957 (local address 127.25.109.1:45541)
0924 16:02:26.772792 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:26.772810 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:26.772922 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:26.773311 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:26.773315 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:26.773335 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:26.773496 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:26.773504 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.774305 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.774308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:26.775233 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:26.775242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.775918 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.775922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:26.776044 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:26.776072 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 16:02:26.776167 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 16:02:26.776243 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":57}
I20260924 16:02:26.776496 1088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.772699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:44747 (local address 127.25.109.3:41657)
0924 16:02:26.772844 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:26.772849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:26.773136 (+ 287us) server_negotiation.cc:407] Connection header received
0924 16:02:26.773281 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:26.773284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:26.773339 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:26.773417 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:26.773693 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.774185 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.775383 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.775795 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.775933 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:26.775993 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:26.776237 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 16:02:26.776334 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:02:26.777117 1105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.772646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:41657 (local address 127.25.109.1:44747)
0924 16:02:26.773023 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:26.773036 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:26.773164 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:26.773445 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:26.773447 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:26.773462 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:26.773569 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:26.773579 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.774300 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.774306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:26.775241 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:26.775251 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.775938 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.775942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:26.776073 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:26.776103 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 16:02:26.776845 (+ 742us) client_negotiation.cc:240] Negotiation successful
0924 16:02:26.776933 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":80,"thread_start_us":129,"threads_started":1}
I20260924 16:02:26.777047 809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c4c81cddfb141ab8b69c5f778278710" candidate_uuid: "c3fa4c2ab3d94a049605e22b6492d8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f758e6c10f3448cdb5d1e3ba31d30925" is_pre_election: true
I20260924 16:02:26.777318 809 raft_consensus.cc:2454] T 8c4c81cddfb141ab8b69c5f778278710 P f758e6c10f3448cdb5d1e3ba31d30925 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3fa4c2ab3d94a049605e22b6492d8ab in term 0.
I20260924 16:02:26.777693 945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c4c81cddfb141ab8b69c5f778278710" candidate_uuid: "c3fa4c2ab3d94a049605e22b6492d8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c9c5af925794483afe9f26d1a0f4291" is_pre_election: true
I20260924 16:02:26.777817 945 raft_consensus.cc:2454] T 8c4c81cddfb141ab8b69c5f778278710 P 1c9c5af925794483afe9f26d1a0f4291 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3fa4c2ab3d94a049605e22b6492d8ab in term 0.
I20260924 16:02:26.777792 607 leader_election.cc:302] T 8c4c81cddfb141ab8b69c5f778278710 P c3fa4c2ab3d94a049605e22b6492d8ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3fa4c2ab3d94a049605e22b6492d8ab, f758e6c10f3448cdb5d1e3ba31d30925; no voters:
I20260924 16:02:26.778093 1103 raft_consensus.cc:2781] T 8c4c81cddfb141ab8b69c5f778278710 P c3fa4c2ab3d94a049605e22b6492d8ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:02:26.778159 1103 raft_consensus.cc:482] T 8c4c81cddfb141ab8b69c5f778278710 P c3fa4c2ab3d94a049605e22b6492d8ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:02:26.778188 1103 raft_consensus.cc:3037] T 8c4c81cddfb141ab8b69c5f778278710 P c3fa4c2ab3d94a049605e22b6492d8ab [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:26.779220 1103 raft_consensus.cc:504] T 8c4c81cddfb141ab8b69c5f778278710 P c3fa4c2ab3d94a049605e22b6492d8ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c9c5af925794483afe9f26d1a0f4291" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41657 } } peers { permanent_uuid: "f758e6c10f3448cdb5d1e3ba31d30925" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38957 } } peers { permanent_uuid: "c3fa4c2ab3d94a049605e22b6492d8ab" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46785 } }
I20260924 16:02:26.779593 1103 leader_election.cc:288] T 8c4c81cddfb141ab8b69c5f778278710 P c3fa4c2ab3d94a049605e22b6492d8ab [CANDIDATE]: Term 1 election: Requested vote from peers 1c9c5af925794483afe9f26d1a0f4291 (127.25.109.3:41657), f758e6c10f3448cdb5d1e3ba31d30925 (127.25.109.2:38957)
I20260924 16:02:26.779929 945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c4c81cddfb141ab8b69c5f778278710" candidate_uuid: "c3fa4c2ab3d94a049605e22b6492d8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c9c5af925794483afe9f26d1a0f4291"
I20260924 16:02:26.779944 809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c4c81cddfb141ab8b69c5f778278710" candidate_uuid: "c3fa4c2ab3d94a049605e22b6492d8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f758e6c10f3448cdb5d1e3ba31d30925"
I20260924 16:02:26.780064 809 raft_consensus.cc:3037] T 8c4c81cddfb141ab8b69c5f778278710 P f758e6c10f3448cdb5d1e3ba31d30925 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:26.780059 945 raft_consensus.cc:3037] T 8c4c81cddfb141ab8b69c5f778278710 P 1c9c5af925794483afe9f26d1a0f4291 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:26.781097 809 raft_consensus.cc:2454] T 8c4c81cddfb141ab8b69c5f778278710 P f758e6c10f3448cdb5d1e3ba31d30925 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3fa4c2ab3d94a049605e22b6492d8ab in term 1.
I20260924 16:02:26.781126 945 raft_consensus.cc:2454] T 8c4c81cddfb141ab8b69c5f778278710 P 1c9c5af925794483afe9f26d1a0f4291 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3fa4c2ab3d94a049605e22b6492d8ab in term 1.
I20260924 16:02:26.781370 607 leader_election.cc:302] T 8c4c81cddfb141ab8b69c5f778278710 P c3fa4c2ab3d94a049605e22b6492d8ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3fa4c2ab3d94a049605e22b6492d8ab, f758e6c10f3448cdb5d1e3ba31d30925; no voters:
I20260924 16:02:26.781560 1103 raft_consensus.cc:2781] T 8c4c81cddfb141ab8b69c5f778278710 P c3fa4c2ab3d94a049605e22b6492d8ab [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:26.781795 1103 raft_consensus.cc:686] T 8c4c81cddfb141ab8b69c5f778278710 P c3fa4c2ab3d94a049605e22b6492d8ab [term 1 LEADER]: Becoming Leader. State: Replica: c3fa4c2ab3d94a049605e22b6492d8ab, State: Running, Role: LEADER
I20260924 16:02:26.782066 1103 consensus_queue.cc:237] T 8c4c81cddfb141ab8b69c5f778278710 P c3fa4c2ab3d94a049605e22b6492d8ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c9c5af925794483afe9f26d1a0f4291" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41657 } } peers { permanent_uuid: "f758e6c10f3448cdb5d1e3ba31d30925" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38957 } } peers { permanent_uuid: "c3fa4c2ab3d94a049605e22b6492d8ab" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46785 } }
I20260924 16:02:26.782524 1105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.782261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.25.109.1:41897)
0924 16:02:26.782406 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:26.782457 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260924 16:02:26.784772 536 catalog_manager.cc:5495] T 8c4c81cddfb141ab8b69c5f778278710 P c3fa4c2ab3d94a049605e22b6492d8ab reported cstate change: term changed from 0 to 1, leader changed from <none> to c3fa4c2ab3d94a049605e22b6492d8ab (127.25.109.1). New cstate: current_term: 1 leader_uuid: "c3fa4c2ab3d94a049605e22b6492d8ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c9c5af925794483afe9f26d1a0f4291" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f758e6c10f3448cdb5d1e3ba31d30925" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3fa4c2ab3d94a049605e22b6492d8ab" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46785 } health_report { overall_health: HEALTHY } } }
W20260924 16:02:26.798228 858 tablet.cc:2369] T 8c4c81cddfb141ab8b69c5f778278710 P f758e6c10f3448cdb5d1e3ba31d30925: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:02:26.807260 721 tablet.cc:2369] T 8c4c81cddfb141ab8b69c5f778278710 P c3fa4c2ab3d94a049605e22b6492d8ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:02:26.809682 1092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.805985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56160 (local address 127.25.109.1:46785)
0924 16:02:26.806139 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:26.806144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:26.806161 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:02:26.806306 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:26.806309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:26.806367 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:26.806484 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:02:26.806755 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.807273 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.808128 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:26.808348 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:26.808520 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:26.808807 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:02:26.809270 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:02:26.809401 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:26.809476 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 16:02:26.809536 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 16:02:26.812719 945 raft_consensus.cc:1261] T 8c4c81cddfb141ab8b69c5f778278710 P 1c9c5af925794483afe9f26d1a0f4291 [term 1 FOLLOWER]: Refusing update from remote peer c3fa4c2ab3d94a049605e22b6492d8ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:02:26.813027 809 raft_consensus.cc:1261] T 8c4c81cddfb141ab8b69c5f778278710 P f758e6c10f3448cdb5d1e3ba31d30925 [term 1 FOLLOWER]: Refusing update from remote peer c3fa4c2ab3d94a049605e22b6492d8ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:02:26.813310 1106 consensus_queue.cc:1035] T 8c4c81cddfb141ab8b69c5f778278710 P c3fa4c2ab3d94a049605e22b6492d8ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c9c5af925794483afe9f26d1a0f4291" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:26.813591 1103 consensus_queue.cc:1035] T 8c4c81cddfb141ab8b69c5f778278710 P c3fa4c2ab3d94a049605e22b6492d8ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "f758e6c10f3448cdb5d1e3ba31d30925" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:26.816080 1101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.815212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.25.109.3:46099)
0924 16:02:26.815905 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:26.815988 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":607}
I20260924 16:02:26.816691 1102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.816366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.25.109.2:33241)
0924 16:02:26.816548 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:26.816610 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260924 16:02:26.819805 1105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.819293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.25.109.1:53247)
0924 16:02:26.819664 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:26.819724 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294}
I20260924 16:02:26.820564 1102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.820265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.25.109.2:35891)
0924 16:02:26.820430 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:26.820489 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 16:02:26.821254 1101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.820959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33903 (local address 127.25.109.3:43593)
0924 16:02:26.821124 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:26.821181 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260924 16:02:26.821628 1110 mvcc.cc:205] Tried to move back new op lower bound from 7332928498939080704 to 7332928498822475776. Current Snapshot: MvccSnapshot[applied={T|T < 7332928498939080704}]
I20260924 16:02:26.830426 537 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:02:26.830619 537 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:02:26.835139 537 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9450614f7a7c4bcebf504f2db67ba64e: Sending DeleteTablet for 3 replicas of tablet 8c4c81cddfb141ab8b69c5f778278710
I20260924 16:02:26.836146 925 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c4c81cddfb141ab8b69c5f778278710 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:26 UTC) from {username='oryx', principal='oryx/127.25.109.60@KRBTEST.COM'} at 127.0.0.1:38336
I20260924 16:02:26.836577 789 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c4c81cddfb141ab8b69c5f778278710 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:26 UTC) from {username='oryx', principal='oryx/127.25.109.60@KRBTEST.COM'} at 127.0.0.1:43148
I20260924 16:02:26.836962 1120 tablet_replica.cc:330] T 8c4c81cddfb141ab8b69c5f778278710 P f758e6c10f3448cdb5d1e3ba31d30925: stopping tablet replica
I20260924 16:02:26.836963 1119 tablet_replica.cc:330] T 8c4c81cddfb141ab8b69c5f778278710 P 1c9c5af925794483afe9f26d1a0f4291: stopping tablet replica
I20260924 16:02:26.837061 651 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c4c81cddfb141ab8b69c5f778278710 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:26 UTC) from {username='oryx', principal='oryx/127.25.109.60@KRBTEST.COM'} at 127.0.0.1:56150
I20260924 16:02:26.837110 1120 raft_consensus.cc:2229] T 8c4c81cddfb141ab8b69c5f778278710 P f758e6c10f3448cdb5d1e3ba31d30925 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:26.837113 1119 raft_consensus.cc:2229] T 8c4c81cddfb141ab8b69c5f778278710 P 1c9c5af925794483afe9f26d1a0f4291 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:26.837255 1120 raft_consensus.cc:2258] T 8c4c81cddfb141ab8b69c5f778278710 P f758e6c10f3448cdb5d1e3ba31d30925 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:26.837276 1119 raft_consensus.cc:2258] T 8c4c81cddfb141ab8b69c5f778278710 P 1c9c5af925794483afe9f26d1a0f4291 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:26.838274 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 500
I20260924 16:02:26.838881 1119 ts_tablet_manager.cc:1905] T 8c4c81cddfb141ab8b69c5f778278710 P 1c9c5af925794483afe9f26d1a0f4291: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:26.839136 1120 ts_tablet_manager.cc:1905] T 8c4c81cddfb141ab8b69c5f778278710 P f758e6c10f3448cdb5d1e3ba31d30925: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:26.841718 1120 ts_tablet_manager.cc:1918] T 8c4c81cddfb141ab8b69c5f778278710 P f758e6c10f3448cdb5d1e3ba31d30925: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:02:26.841718 1119 ts_tablet_manager.cc:1918] T 8c4c81cddfb141ab8b69c5f778278710 P 1c9c5af925794483afe9f26d1a0f4291: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:02:26.841809 1120 log.cc:1196] T 8c4c81cddfb141ab8b69c5f778278710 P f758e6c10f3448cdb5d1e3ba31d30925: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-1/wal/wals/8c4c81cddfb141ab8b69c5f778278710
I20260924 16:02:26.841869 1119 log.cc:1196] T 8c4c81cddfb141ab8b69c5f778278710 P 1c9c5af925794483afe9f26d1a0f4291: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265708917108-26036-0/minicluster-data/ts-2/wal/wals/8c4c81cddfb141ab8b69c5f778278710
I20260924 16:02:26.842166 1119 ts_tablet_manager.cc:1939] T 8c4c81cddfb141ab8b69c5f778278710 P 1c9c5af925794483afe9f26d1a0f4291: Deleting consensus metadata
I20260924 16:02:26.842168 1120 ts_tablet_manager.cc:1939] T 8c4c81cddfb141ab8b69c5f778278710 P f758e6c10f3448cdb5d1e3ba31d30925: Deleting consensus metadata
I20260924 16:02:26.842936 525 catalog_manager.cc:4841] TS f758e6c10f3448cdb5d1e3ba31d30925 (127.25.109.2:38957): tablet 8c4c81cddfb141ab8b69c5f778278710 (table test-table [id=a7745e2bb4bf48ccb9478d1ffdc32794]) successfully deleted
I20260924 16:02:26.842936 522 catalog_manager.cc:4841] TS 1c9c5af925794483afe9f26d1a0f4291 (127.25.109.3:41657): tablet 8c4c81cddfb141ab8b69c5f778278710 (table test-table [id=a7745e2bb4bf48ccb9478d1ffdc32794]) successfully deleted
I20260924 16:02:26.846517 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 726
I20260924 16:02:26.846784 1087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:26.846437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:46785 (local address 127.0.0.1:56168)
0924 16:02:26.846604 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:26.846674 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.1:46785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 16:02:26.847000 522 catalog_manager.cc:4584] TS c3fa4c2ab3d94a049605e22b6492d8ab (127.25.109.1:46785): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8c4c81cddfb141ab8b69c5f778278710: Network error: Client connection negotiation failed: client connection to 127.25.109.1:46785: connect: Connection refused (error 111)
I20260924 16:02:26.854532 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 862
I20260924 16:02:26.860807 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 32686
I20260924 16:02:26.867280 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 320
2026-09-24T16:02:26Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7581 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:02:26 dist-test-slave-n106 krb5kdc[1124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:02:26 dist-test-slave-n106 krb5kdc[1124](info): set up 2 sockets
Sep 24 16:02:26 dist-test-slave-n106 krb5kdc[1124](info): commencing operation
krb5kdc: starting...
W20260924 16:02:27.938253 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:02:28 dist-test-slave-n106 krb5kdc[1124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:02:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:02:28Z Disabled control of system clock
WARNING: no policy specified for oryx/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.62.keytab.
Entry for principal oryx/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.62.keytab.
I20260924 16:02:28.059887 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:46829
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.62.keytab
--principal=oryx/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:46829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:28.175263 1138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:28.175616 1138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:28.175701 1138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:28.179829 1138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:02:28.179930 1138 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 16:02:28.179963 1138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:28.180006 1138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:28.180045 1138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:02:28.180096 1138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:02:28.185745 1138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:46829
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.62.keytab
--principal=oryx/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:46829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:28.186926 1138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:28.188042 1138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:02:28.197324 1138 server_base.cc:1034] running on GCE node
W20260924 16:02:28.197304 1144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:28.197314 1143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:28.197342 1146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:28.198151 1138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:28.199329 1138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:28.200479 1138 hybrid_clock.cc:648] HybridClock initialized: now 1790265748200460 us; error 38 us; skew 500 ppm
Sep 24 16:02:28 dist-test-slave-n106 krb5kdc[1124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265748, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:28.203279 1138 init.cc:377] Logged in from keytab as oryx/127.25.109.62@KRBTEST.COM (short username oryx)
I20260924 16:02:28.204191 1138 webserver.cc:460] Webserver started at http://127.25.109.62:36197/ using document root <none> and password file <none>
I20260924 16:02:28.204824 1138 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:28.204911 1138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:28.205148 1138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:28.207330 1138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "8c62fa2dda514a4297a2333220ca9ee2"
format_stamp: "Formatted at 2026-09-24 16:02:28 on dist-test-slave-n106"
server_key: "b491b8b94a445b4621801495dbd1758a"
server_key_iv: "d7d3c2aaffc902b3057620143dbef80c"
server_key_version: "encryptionkey@0"
I20260924 16:02:28.207917 1138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "8c62fa2dda514a4297a2333220ca9ee2"
format_stamp: "Formatted at 2026-09-24 16:02:28 on dist-test-slave-n106"
server_key: "b491b8b94a445b4621801495dbd1758a"
server_key_iv: "d7d3c2aaffc902b3057620143dbef80c"
server_key_version: "encryptionkey@0"
I20260924 16:02:28.211864 1138 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260924 16:02:28.214484 1153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:28.215744 1138 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:02:28.215871 1138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "8c62fa2dda514a4297a2333220ca9ee2"
format_stamp: "Formatted at 2026-09-24 16:02:28 on dist-test-slave-n106"
server_key: "b491b8b94a445b4621801495dbd1758a"
server_key_iv: "d7d3c2aaffc902b3057620143dbef80c"
server_key_version: "encryptionkey@0"
I20260924 16:02:28.215982 1138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:28.244096 1138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:28.247786 1138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:28.248049 1138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:28.256273 1138 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:46829
I20260924 16:02:28.256335 1216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:46829 every 8 connection(s)
I20260924 16:02:28.257356 1138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:02:28.260272 1217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:28.266122 1217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c62fa2dda514a4297a2333220ca9ee2: Bootstrap starting.
I20260924 16:02:28.266582 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 1138
I20260924 16:02:28.266733 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:02:28.267007 26036 external_mini_cluster.cc:1428] Setting key 9ebb9293606e716c0baa3ebff1fb5fa0
I20260924 16:02:28.269019 1217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c62fa2dda514a4297a2333220ca9ee2: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:28.269932 1217 log.cc:824] T 00000000000000000000000000000000 P 8c62fa2dda514a4297a2333220ca9ee2: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:28.271838 1217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c62fa2dda514a4297a2333220ca9ee2: No bootstrap required, opened a new log
I20260924 16:02:28.275290 1217 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c62fa2dda514a4297a2333220ca9ee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c62fa2dda514a4297a2333220ca9ee2" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 46829 } }
I20260924 16:02:28.275542 1217 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c62fa2dda514a4297a2333220ca9ee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:28.275648 1217 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c62fa2dda514a4297a2333220ca9ee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c62fa2dda514a4297a2333220ca9ee2, State: Initialized, Role: FOLLOWER
Sep 24 16:02:28 dist-test-slave-n106 krb5kdc[1124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM
I20260924 16:02:28.276180 1217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c62fa2dda514a4297a2333220ca9ee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c62fa2dda514a4297a2333220ca9ee2" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 46829 } }
I20260924 16:02:28.276351 1217 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c62fa2dda514a4297a2333220ca9ee2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:02:28.276443 1217 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c62fa2dda514a4297a2333220ca9ee2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:02:28.276554 1217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c62fa2dda514a4297a2333220ca9ee2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:28.277640 1217 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c62fa2dda514a4297a2333220ca9ee2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c62fa2dda514a4297a2333220ca9ee2" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 46829 } }
I20260924 16:02:28.278033 1217 leader_election.cc:302] T 00000000000000000000000000000000 P 8c62fa2dda514a4297a2333220ca9ee2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c62fa2dda514a4297a2333220ca9ee2; no voters:
I20260924 16:02:28.278405 1217 leader_election.cc:288] T 00000000000000000000000000000000 P 8c62fa2dda514a4297a2333220ca9ee2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:02:28.278517 1222 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c62fa2dda514a4297a2333220ca9ee2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:28.278726 1222 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c62fa2dda514a4297a2333220ca9ee2 [term 1 LEADER]: Becoming Leader. State: Replica: 8c62fa2dda514a4297a2333220ca9ee2, State: Running, Role: LEADER
I20260924 16:02:28.279137 1222 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c62fa2dda514a4297a2333220ca9ee2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c62fa2dda514a4297a2333220ca9ee2" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 46829 } }
I20260924 16:02:28.279794 1217 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c62fa2dda514a4297a2333220ca9ee2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:02:28.281507 1224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c62fa2dda514a4297a2333220ca9ee2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c62fa2dda514a4297a2333220ca9ee2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c62fa2dda514a4297a2333220ca9ee2" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 46829 } } }
I20260924 16:02:28.281661 1224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c62fa2dda514a4297a2333220ca9ee2 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:28.282575 1220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:28.268379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34756 (local address 127.25.109.62:46829)
0924 16:02:28.268885 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:28.268907 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:28.268956 (+ 49us) server_negotiation.cc:407] Connection header received
0924 16:02:28.269867 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:28.269892 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:28.270341 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:28.270711 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:28.271531 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:28.272655 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:28.273503 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:28.273993 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:28.274161 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:28.276969 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:28.276976 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:28.277002 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:28.277031 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:28.279086 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:28.279593 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:28.279599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:28.279605 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:28.279748 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:28.280051 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:28.280054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:28.280056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:28.280666 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:28.280877 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:28.281157 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 16:02:28.281476 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":126,"threads_started":1}
I20260924 16:02:28.283093 1225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c62fa2dda514a4297a2333220ca9ee2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c62fa2dda514a4297a2333220ca9ee2. Latest consensus state: current_term: 1 leader_uuid: "8c62fa2dda514a4297a2333220ca9ee2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c62fa2dda514a4297a2333220ca9ee2" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 46829 } } }
I20260924 16:02:28.283265 1225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c62fa2dda514a4297a2333220ca9ee2 [sys.catalog]: This master's current role is: LEADER
W20260924 16:02:28.284955 1238 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8c62fa2dda514a4297a2333220ca9ee2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:02:28.285045 1238 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:02:28.285104 1237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:02:28.285808 1237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:02:28.291352 1237 catalog_manager.cc:1347] Generated new cluster ID: 89177a26bd4b4a5f90c4b0c8ba3d27b2
I20260924 16:02:28.291427 1237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:02:28.314152 1237 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:02:28.315177 1237 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:02:28.321161 1237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c62fa2dda514a4297a2333220ca9ee2: Generated new TSK 0
I20260924 16:02:28.321931 1237 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.25.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.1.keytab.
Entry for principal oryx/127.25.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.1.keytab.
I20260924 16:02:28.353219 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:0
--local_ip_for_outbound_sockets=127.25.109.1
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:46829
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.1.keytab
--principal=oryx/127.25.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:28.474781 1244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:28.475081 1244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:28.475128 1244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:28.479425 1244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:28.479483 1244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:28.479527 1244 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:02:28.479602 1244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:02:28.486022 1244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.1.keytab
--principal=oryx/127.25.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.109.62:46829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:28.487251 1244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:28.488461 1244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:28.499277 1250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:28.499413 1249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:28.499320 1244 server_base.cc:1034] running on GCE node
W20260924 16:02:28.499277 1252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:28.500253 1244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:28.500969 1244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:28.502162 1244 hybrid_clock.cc:648] HybridClock initialized: now 1790265748502143 us; error 31 us; skew 500 ppm
Sep 24 16:02:28 dist-test-slave-n106 krb5kdc[1124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265748, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:28.505074 1244 init.cc:377] Logged in from keytab as oryx/127.25.109.1@KRBTEST.COM (short username oryx)
I20260924 16:02:28.506011 1244 webserver.cc:460] Webserver started at http://127.25.109.1:44257/ using document root <none> and password file <none>
I20260924 16:02:28.506604 1244 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:28.506716 1244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:28.506968 1244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:28.509263 1244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-0/data/instance:
uuid: "fd528086b31748cfa35d0ec7e5012efd"
format_stamp: "Formatted at 2026-09-24 16:02:28 on dist-test-slave-n106"
server_key: "3bbd9c6ccc731622fb55dfd35a337a8b"
server_key_iv: "880396d4ed2088acf7b71a464f620361"
server_key_version: "encryptionkey@0"
I20260924 16:02:28.509871 1244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance:
uuid: "fd528086b31748cfa35d0ec7e5012efd"
format_stamp: "Formatted at 2026-09-24 16:02:28 on dist-test-slave-n106"
server_key: "3bbd9c6ccc731622fb55dfd35a337a8b"
server_key_iv: "880396d4ed2088acf7b71a464f620361"
server_key_version: "encryptionkey@0"
I20260924 16:02:28.513785 1244 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:02:28.516394 1259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:28.517735 1244 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:02:28.517869 1244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "fd528086b31748cfa35d0ec7e5012efd"
format_stamp: "Formatted at 2026-09-24 16:02:28 on dist-test-slave-n106"
server_key: "3bbd9c6ccc731622fb55dfd35a337a8b"
server_key_iv: "880396d4ed2088acf7b71a464f620361"
server_key_version: "encryptionkey@0"
I20260924 16:02:28.517988 1244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:28.528098 1244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:28.531394 1244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:28.531600 1244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:28.533213 1244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:28.533278 1244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:28.533346 1244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:28.533391 1244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:02:28 dist-test-slave-n106 krb5kdc[1124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265748, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM
I20260924 16:02:28.544283 1244 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:36345
I20260924 16:02:28.544341 1376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:36345 every 8 connection(s)
I20260924 16:02:28.545320 1244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:02:28.548527 1220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:28.534053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:60399 (local address 127.25.109.62:46829)
0924 16:02:28.534185 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:28.534189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:28.535283 (+ 1094us) server_negotiation.cc:407] Connection header received
0924 16:02:28.536735 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:28.536739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:28.536812 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:28.536884 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:28.538858 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:28.539845 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:28.541360 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:28.541646 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:28.541774 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:28.544784 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:28.544788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:28.544790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:28.544820 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:28.546563 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:28.547023 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:28.547026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:28.547027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:28.547115 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:28.547403 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:28.547405 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:28.547406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:28.547637 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:28.547722 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:28.548176 (+ 454us) server_negotiation.cc:299] Negotiation successful
0924 16:02:28.548293 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 16:02:28.549336 1268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:28.534600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:46829 (local address 127.25.109.1:60399)
0924 16:02:28.535116 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:28.535161 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:28.536319 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:28.537368 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:28.537375 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:28.537946 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:28.538667 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:28.538684 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:28.539996 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:28.540014 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:28.541212 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:28.541223 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:28.541818 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:28.541822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:28.542027 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:28.542748 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:28.542765 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:28.544603 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:28.546671 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:28.546675 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:28.546692 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:28.546936 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:28.547193 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:28.547195 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:28.547196 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:28.547321 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:28.547726 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:28.547731 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:28.547949 (+ 218us) client_negotiation.cc:769] Sending connection context
0924 16:02:28.548096 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 16:02:28.548350 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":141,"threads_started":1}
I20260924 16:02:28.550137 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 1244
I20260924 16:02:28.550279 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance
I20260924 16:02:28.550649 26036 external_mini_cluster.cc:1428] Setting key 1197b646e6593c08d17ff5f9701950a1
I20260924 16:02:28.551000 1377 heartbeater.cc:347] Connected to a master server at 127.25.109.62:46829
I20260924 16:02:28.551164 1377 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:28.551849 1377 heartbeater.cc:511] Master 127.25.109.62:46829 requested a full tablet report, sending...
I20260924 16:02:28.553644 1170 ts_manager.cc:194] Registered new tserver with Master: fd528086b31748cfa35d0ec7e5012efd (127.25.109.1:36345)
I20260924 16:02:28.555003 1170 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.109.1@KRBTEST.COM'} at 127.25.109.1:60399
WARNING: no policy specified for oryx/127.25.109.2@KRBTEST.COM; defaulting to no policy
I20260924 16:02:28.562139 1220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:28.553397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:35295 (local address 127.25.109.62:46829)
0924 16:02:28.553535 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:28.553540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:28.553854 (+ 314us) server_negotiation.cc:407] Connection header received
0924 16:02:28.553901 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:28.553906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:28.553963 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:28.554047 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:28.554735 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:28.555428 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:28.556794 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:28.557101 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:28.557220 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:28.558503 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:28.558510 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:28.558514 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:28.558547 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:28.560293 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:28.560808 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:28.560812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:28.560816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:28.560912 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:28.561181 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:28.561187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:28.561192 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:28.561423 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:28.561523 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:28.561844 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 16:02:28.561977 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 16:02:28.562151 1383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:28.553347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:46829 (local address 127.25.109.1:35295)
0924 16:02:28.553681 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:28.553693 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:28.553804 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:28.554184 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:28.554189 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:28.554436 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:28.554600 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:28.554609 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:28.555567 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:28.555574 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:28.556664 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:28.556680 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:28.557245 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:28.557251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:28.557364 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:28.557963 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:28.557977 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:28.558350 (+ 373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:28.560487 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:28.560495 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:28.560496 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:28.560706 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:28.561009 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:28.561011 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:28.561013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:28.561090 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:28.561544 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:28.561547 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:28.561632 (+ 85us) client_negotiation.cc:769] Sending connection context
0924 16:02:28.561780 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 16:02:28.561938 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.2.keytab.
Entry for principal oryx/127.25.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.2.keytab.
I20260924 16:02:28.579762 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:0
--local_ip_for_outbound_sockets=127.25.109.2
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:46829
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.2.keytab
--principal=oryx/127.25.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:28.692250 1386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:28.692534 1386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:28.692611 1386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:28.696627 1386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:28.696681 1386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:28.696756 1386 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:02:28.696828 1386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:02:28.702015 1386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.2.keytab
--principal=oryx/127.25.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.109.62:46829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:28.703088 1386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:28.704152 1386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:28.713920 1391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:28.713948 1392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:28.713966 1394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:28.714135 1386 server_base.cc:1034] running on GCE node
I20260924 16:02:28.714632 1386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:28.715271 1386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:28.716487 1386 hybrid_clock.cc:648] HybridClock initialized: now 1790265748716459 us; error 32 us; skew 500 ppm
Sep 24 16:02:28 dist-test-slave-n106 krb5kdc[1124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265748, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:28.719197 1386 init.cc:377] Logged in from keytab as oryx/127.25.109.2@KRBTEST.COM (short username oryx)
I20260924 16:02:28.720168 1386 webserver.cc:460] Webserver started at http://127.25.109.2:35149/ using document root <none> and password file <none>
I20260924 16:02:28.720824 1386 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:28.720886 1386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:28.721057 1386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:28.723279 1386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-1/data/instance:
uuid: "014fd818c4b34521b5e2587cf075b95a"
format_stamp: "Formatted at 2026-09-24 16:02:28 on dist-test-slave-n106"
server_key: "2bc5a80b0704800d1d73fbea43408887"
server_key_iv: "f2fc99223eb15a0dcb7c54d17006bacf"
server_key_version: "encryptionkey@0"
I20260924 16:02:28.723835 1386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance:
uuid: "014fd818c4b34521b5e2587cf075b95a"
format_stamp: "Formatted at 2026-09-24 16:02:28 on dist-test-slave-n106"
server_key: "2bc5a80b0704800d1d73fbea43408887"
server_key_iv: "f2fc99223eb15a0dcb7c54d17006bacf"
server_key_version: "encryptionkey@0"
I20260924 16:02:28.727710 1386 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 16:02:28.730273 1401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:28.731515 1386 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:02:28.731639 1386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "014fd818c4b34521b5e2587cf075b95a"
format_stamp: "Formatted at 2026-09-24 16:02:28 on dist-test-slave-n106"
server_key: "2bc5a80b0704800d1d73fbea43408887"
server_key_iv: "f2fc99223eb15a0dcb7c54d17006bacf"
server_key_version: "encryptionkey@0"
I20260924 16:02:28.731755 1386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:28.745018 1386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:28.748148 1386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:28.748313 1386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:28.749881 1386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:28.749935 1386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:28.750001 1386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:28.750034 1386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:02:28 dist-test-slave-n106 krb5kdc[1124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265748, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM
I20260924 16:02:28.761518 1386 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:35349
I20260924 16:02:28.761581 1516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:35349 every 8 connection(s)
I20260924 16:02:28.762774 1386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:02:28.764822 1220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:28.751094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:33535 (local address 127.25.109.62:46829)
0924 16:02:28.751233 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:28.751238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:28.752268 (+ 1030us) server_negotiation.cc:407] Connection header received
0924 16:02:28.753495 (+ 1227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:28.753499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:28.753567 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:28.753673 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:28.755143 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:28.755693 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:28.757007 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:28.757288 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:28.757421 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:28.760725 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:28.760729 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:28.760732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:28.760766 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:28.762497 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:28.763194 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:28.763198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:28.763200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:28.763293 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:28.763755 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:28.763761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:28.763766 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:28.763992 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:28.764070 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:28.764589 (+ 519us) server_negotiation.cc:299] Negotiation successful
0924 16:02:28.764697 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:02:28.765692 1410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:28.751592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:46829 (local address 127.25.109.2:33535)
0924 16:02:28.752098 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:28.752154 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:28.753249 (+ 1095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:28.753899 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:28.753911 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:28.754379 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:28.754973 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:28.754988 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:28.755829 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:28.755833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:28.756864 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:28.756875 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:28.757487 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:28.757491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:28.757665 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:28.758363 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:28.758386 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:28.760528 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:28.762665 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:28.762670 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:28.762684 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:28.763033 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:28.763458 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:28.763461 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:28.763463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:28.763640 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:28.764076 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:28.764081 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:28.764315 (+ 234us) client_negotiation.cc:769] Sending connection context
0924 16:02:28.764534 (+ 219us) client_negotiation.cc:240] Negotiation successful
0924 16:02:28.764809 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":140,"threads_started":1}
I20260924 16:02:28.766394 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 1386
I20260924 16:02:28.766535 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance
I20260924 16:02:28.766862 26036 external_mini_cluster.cc:1428] Setting key 01ef82212d2eaa273759d1c0696aa2ad
I20260924 16:02:28.767043 1517 heartbeater.cc:347] Connected to a master server at 127.25.109.62:46829
I20260924 16:02:28.767191 1517 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:28.767794 1517 heartbeater.cc:511] Master 127.25.109.62:46829 requested a full tablet report, sending...
I20260924 16:02:28.768891 1168 ts_manager.cc:194] Registered new tserver with Master: 014fd818c4b34521b5e2587cf075b95a (127.25.109.2:35349)
I20260924 16:02:28.769485 1168 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.109.2@KRBTEST.COM'} at 127.25.109.2:33535
WARNING: no policy specified for oryx/127.25.109.3@KRBTEST.COM; defaulting to no policy
I20260924 16:02:28.778393 1220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:28.770843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:37261 (local address 127.25.109.62:46829)
0924 16:02:28.770947 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:28.770952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:28.770966 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:02:28.771044 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:28.771047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:28.771087 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:28.771174 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:28.771766 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:28.772292 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:28.773636 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:28.773845 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:28.773926 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:28.775301 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:28.775304 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:28.775306 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:28.775327 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:28.776786 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:28.777281 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:28.777284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:28.777285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:28.777361 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:28.777614 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:28.777616 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:28.777618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:28.777826 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:28.777904 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:28.778173 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 16:02:28.778267 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 16:02:28.778399 1524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:28.770525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:46829 (local address 127.25.109.2:37261)
0924 16:02:28.770837 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:28.770856 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:28.770956 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:28.771215 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:28.771218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:28.771472 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:28.771627 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:28.771638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:28.772446 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:28.772459 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:28.773513 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:28.773529 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:28.773964 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:28.773970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:28.774107 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:28.774656 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:28.774676 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:28.775150 (+ 474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:28.776891 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:28.776895 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:28.776897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:28.777179 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:28.777453 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:28.777456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:28.777457 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:28.777538 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:28.777911 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:28.777913 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:28.778039 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 16:02:28.778124 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 16:02:28.778243 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":94,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.3.keytab.
Entry for principal oryx/127.25.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.3.keytab.
I20260924 16:02:28.796599 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:0
--local_ip_for_outbound_sockets=127.25.109.3
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:46829
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.3.keytab
--principal=oryx/127.25.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:28.911540 1526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:28.911821 1526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:28.911867 1526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:28.915956 1526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:28.916015 1526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:28.916054 1526 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:02:28.916088 1526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:02:28.921674 1526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.3.keytab
--principal=oryx/127.25.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.109.62:46829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:28.922830 1526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:28.923919 1526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:28.934427 1534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:28.934407 1526 server_base.cc:1034] running on GCE node
W20260924 16:02:28.934374 1531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:28.934370 1532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:28.935155 1526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:28.935959 1526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:28.937151 1526 hybrid_clock.cc:648] HybridClock initialized: now 1790265748937133 us; error 29 us; skew 500 ppm
Sep 24 16:02:28 dist-test-slave-n106 krb5kdc[1124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265748, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:28.940438 1526 init.cc:377] Logged in from keytab as oryx/127.25.109.3@KRBTEST.COM (short username oryx)
I20260924 16:02:28.941349 1526 webserver.cc:460] Webserver started at http://127.25.109.3:43047/ using document root <none> and password file <none>
I20260924 16:02:28.941947 1526 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:28.942008 1526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:28.942243 1526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:28.944373 1526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-2/data/instance:
uuid: "23a7821f743042fb9fa18864c1487c4c"
format_stamp: "Formatted at 2026-09-24 16:02:28 on dist-test-slave-n106"
server_key: "b42a5de901e6aecbd05b539db1278969"
server_key_iv: "a500e471585fee0bdafb0bc01bc4491d"
server_key_version: "encryptionkey@0"
I20260924 16:02:28.945021 1526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance:
uuid: "23a7821f743042fb9fa18864c1487c4c"
format_stamp: "Formatted at 2026-09-24 16:02:28 on dist-test-slave-n106"
server_key: "b42a5de901e6aecbd05b539db1278969"
server_key_iv: "a500e471585fee0bdafb0bc01bc4491d"
server_key_version: "encryptionkey@0"
I20260924 16:02:28.948916 1526 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:02:28.951541 1541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:28.952991 1526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:02:28.953127 1526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "23a7821f743042fb9fa18864c1487c4c"
format_stamp: "Formatted at 2026-09-24 16:02:28 on dist-test-slave-n106"
server_key: "b42a5de901e6aecbd05b539db1278969"
server_key_iv: "a500e471585fee0bdafb0bc01bc4491d"
server_key_version: "encryptionkey@0"
I20260924 16:02:28.953233 1526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:28.965529 1526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:28.968922 1526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:28.969113 1526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:28.970722 1526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:28.970772 1526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:28.970813 1526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:28.970858 1526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:02:28 dist-test-slave-n106 krb5kdc[1124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265748, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM
I20260924 16:02:28.981777 1526 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:35417
I20260924 16:02:28.981877 1658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:35417 every 8 connection(s)
I20260924 16:02:28.982862 1526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:02:28.986236 1220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:28.971642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:46515 (local address 127.25.109.62:46829)
0924 16:02:28.971767 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:28.971771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:28.972912 (+ 1141us) server_negotiation.cc:407] Connection header received
0924 16:02:28.974067 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:28.974071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:28.974132 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:28.974223 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:28.975904 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:28.976790 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:28.978648 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:28.978870 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:28.978963 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:28.982352 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:28.982356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:28.982359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:28.982391 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:28.984082 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:28.984624 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:28.984627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:28.984628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:28.984714 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:28.985104 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:28.985108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:28.985109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:28.985305 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:28.985390 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:28.985978 (+ 588us) server_negotiation.cc:299] Negotiation successful
0924 16:02:28.986108 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 16:02:28.987017 1550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:28.972186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:46829 (local address 127.25.109.3:46515)
0924 16:02:28.972762 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:28.972803 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:28.973827 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:28.974456 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:28.974467 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:28.974962 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:28.975716 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:28.975740 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:28.976936 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:28.976944 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:28.978512 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:28.978522 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:28.979027 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:28.979031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:28.979259 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:28.979974 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:28.980001 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:28.982173 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:28.984195 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:28.984200 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:28.984212 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:28.984535 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:28.984790 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:28.984792 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:28.984794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:28.984930 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:28.985444 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:28.985450 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:28.985706 (+ 256us) client_negotiation.cc:769] Sending connection context
0924 16:02:28.985867 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 16:02:28.986120 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":157,"threads_started":1}
I20260924 16:02:28.988215 1659 heartbeater.cc:347] Connected to a master server at 127.25.109.62:46829
I20260924 16:02:28.988389 1659 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:28.988919 1659 heartbeater.cc:511] Master 127.25.109.62:46829 requested a full tablet report, sending...
I20260924 16:02:28.989935 1170 ts_manager.cc:194] Registered new tserver with Master: 23a7821f743042fb9fa18864c1487c4c (127.25.109.3:35417)
I20260924 16:02:28.990535 1170 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.109.3@KRBTEST.COM'} at 127.25.109.3:46515
I20260924 16:02:28.993054 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 1526
I20260924 16:02:28.993311 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance
I20260924 16:02:28.993680 26036 external_mini_cluster.cc:1428] Setting key 9e0077c32bcc84e1fa7179b79b0da343
I20260924 16:02:28.995829 26036 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:02:28.997922 1220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:28.990569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:48471 (local address 127.25.109.62:46829)
0924 16:02:28.990761 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:28.990766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:28.990905 (+ 139us) server_negotiation.cc:407] Connection header received
0924 16:02:28.991048 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:28.991051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:28.991095 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:28.991175 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:28.991591 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:28.992125 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:28.993026 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:28.993291 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:28.993400 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:28.994595 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:28.994599 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:28.994602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:28.994626 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:28.996149 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:28.996692 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:28.996696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:28.996697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:28.996780 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:28.997103 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:28.997105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:28.997106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:28.997278 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:28.997354 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:28.997666 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 16:02:28.997771 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260924 16:02:28.998066 1665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:28.990481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:46829 (local address 127.25.109.3:48471)
0924 16:02:28.990829 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:28.990840 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:28.990954 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:28.991190 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:28.991193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:28.991387 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:28.991481 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:28.991486 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:28.992258 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:28.992261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:28.992891 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:28.992901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:28.993404 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:28.993407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:28.993500 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:28.994064 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:28.994084 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:28.994463 (+ 379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:28.996252 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:28.996255 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:28.996257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:28.996579 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:28.996878 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:28.996884 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:28.996890 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:28.997007 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:28.997369 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:28.997375 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:28.997501 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 16:02:28.997617 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 16:02:28.997801 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":103,"threads_started":1}
Sep 24 16:02:29 dist-test-slave-n106 krb5kdc[1124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265749, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 16:02:29 dist-test-slave-n106 krb5kdc[1124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:29 dist-test-slave-n106 krb5kdc[1124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 16:02:29.020891 1220 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:29.014471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34762 (local address 127.25.109.62:46829)
0924 16:02:29.014597 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:29.014600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:29.014612 (+ 12us) server_negotiation.cc:407] Connection header received
0924 16:02:29.014653 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:29.014655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:29.014694 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:29.014781 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:29.015519 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.016147 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.017081 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.017301 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.017408 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:29.020730 (+ 3322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34762: BlockingRecv error: recv got EOF from 127.0.0.1:34762 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 16:02:29 dist-test-slave-n106 krb5kdc[1124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265749, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM
I20260924 16:02:29.030253 1220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:29.022286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34776 (local address 127.25.109.62:46829)
0924 16:02:29.022434 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:29.022438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:29.022557 (+ 119us) server_negotiation.cc:407] Connection header received
0924 16:02:29.022706 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:29.022709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:29.022761 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:29.022843 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:29.023460 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.023961 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.024795 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.025008 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.025113 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:29.027252 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:29.027256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:29.027259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:29.027287 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:29.028665 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:29.029075 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:29.029078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:29.029080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:29.029162 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:29.029419 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:29.029423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:29.029425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:29.029599 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:29.029758 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:29.029969 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 16:02:29.030085 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 16:02:29.033293 1170 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:02:29.036531 1170 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:02:29.051501 1682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:29.045141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45466 (local address 127.25.109.3:35417)
0924 16:02:29.045571 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:29.045582 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:29.045631 (+ 49us) server_negotiation.cc:407] Connection header received
0924 16:02:29.045717 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:29.045723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:29.045955 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:29.046141 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:29.046646 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.047985 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.049883 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.050632 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.050752 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:29.050995 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:29.051241 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 16:02:29.051345 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":113,"threads_started":1}
I20260924 16:02:29.051548 1681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:29.044998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:35417 (local address 127.0.0.1:45466)
0924 16:02:29.045342 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:29.045397 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:29.045519 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:29.046329 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:29.046335 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:29.046360 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:29.046498 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:29.046506 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.048105 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.048108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:29.049742 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:29.049754 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.050782 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.050786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:29.050965 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:29.051055 (+ 90us) client_negotiation.cc:769] Sending connection context
0924 16:02:29.051192 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 16:02:29.051319 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":261,"thread_start_us":84,"threads_started":1}
I20260924 16:02:29.051559 1683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:29.045210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:35349 (local address 127.0.0.1:54844)
0924 16:02:29.046071 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:29.046084 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:29.046177 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:29.047285 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:29.047290 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:29.047309 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:29.047378 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:29.047382 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.048587 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.048590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:29.050033 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:29.050045 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.051199 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.051203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:29.051287 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:29.051308 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 16:02:29.051367 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 16:02:29.051425 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"mutex_wait_us":212,"thread_start_us":85,"threads_started":1}
I20260924 16:02:29.051633 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:29.045797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54844 (local address 127.25.109.2:35349)
0924 16:02:29.046821 (+ 1024us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:29.046834 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:29.046859 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:02:29.046920 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:29.046924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:29.047102 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:29.047487 (+ 385us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:29.047591 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.048482 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.050163 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.051082 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.051200 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:29.051353 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:29.051397 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 16:02:29.051470 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":918,"thread_start_us":106,"threads_started":1}
I20260924 16:02:29.053354 1684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:29.045756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:36345 (local address 127.0.0.1:58766)
0924 16:02:29.046476 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:29.046488 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:29.046568 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:29.047067 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:29.047070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:29.047088 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:29.047198 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:29.047207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.048459 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.048463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:29.050033 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:29.050045 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.053000 (+ 2955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.053004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:29.053097 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:29.053131 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 16:02:29.053194 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 16:02:29.053247 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":193,"thread_start_us":85,"threads_started":1}
I20260924 16:02:29.053622 1686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:29.046173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58766 (local address 127.25.109.1:36345)
0924 16:02:29.046583 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:29.046600 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:29.046637 (+ 37us) server_negotiation.cc:407] Connection header received
0924 16:02:29.046715 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:29.046723 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:29.046920 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:29.047091 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:29.047315 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.048328 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.051825 (+ 3497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.052701 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.052805 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:29.052934 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:29.053252 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 16:02:29.053354 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":76,"threads_started":1}
I20260924 16:02:29.055194 1592 tablet_service.cc:1467] Processing CreateTablet for tablet 546acbf478ac442b88ff0569bf7ea485 (DEFAULT_TABLE table=test-table [id=210865ca29074c7e8532d2de13ec50e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:29.055518 1451 tablet_service.cc:1467] Processing CreateTablet for tablet 546acbf478ac442b88ff0569bf7ea485 (DEFAULT_TABLE table=test-table [id=210865ca29074c7e8532d2de13ec50e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:29.056277 1592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 546acbf478ac442b88ff0569bf7ea485. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:29.056087 1309 tablet_service.cc:1467] Processing CreateTablet for tablet 546acbf478ac442b88ff0569bf7ea485 (DEFAULT_TABLE table=test-table [id=210865ca29074c7e8532d2de13ec50e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:29.056617 1451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 546acbf478ac442b88ff0569bf7ea485. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:29.057129 1309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 546acbf478ac442b88ff0569bf7ea485. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:29.063107 1687 tablet_bootstrap.cc:492] T 546acbf478ac442b88ff0569bf7ea485 P 23a7821f743042fb9fa18864c1487c4c: Bootstrap starting.
I20260924 16:02:29.064431 1688 tablet_bootstrap.cc:492] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a: Bootstrap starting.
I20260924 16:02:29.065356 1689 tablet_bootstrap.cc:492] T 546acbf478ac442b88ff0569bf7ea485 P fd528086b31748cfa35d0ec7e5012efd: Bootstrap starting.
I20260924 16:02:29.066313 1687 tablet_bootstrap.cc:654] T 546acbf478ac442b88ff0569bf7ea485 P 23a7821f743042fb9fa18864c1487c4c: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:29.066720 1688 tablet_bootstrap.cc:654] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:29.067314 1687 log.cc:824] T 546acbf478ac442b88ff0569bf7ea485 P 23a7821f743042fb9fa18864c1487c4c: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:29.067392 1689 tablet_bootstrap.cc:654] T 546acbf478ac442b88ff0569bf7ea485 P fd528086b31748cfa35d0ec7e5012efd: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:29.067644 1688 log.cc:824] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:29.068662 1689 log.cc:824] T 546acbf478ac442b88ff0569bf7ea485 P fd528086b31748cfa35d0ec7e5012efd: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:29.069509 1687 tablet_bootstrap.cc:492] T 546acbf478ac442b88ff0569bf7ea485 P 23a7821f743042fb9fa18864c1487c4c: No bootstrap required, opened a new log
I20260924 16:02:29.069643 1688 tablet_bootstrap.cc:492] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a: No bootstrap required, opened a new log
I20260924 16:02:29.069741 1687 ts_tablet_manager.cc:1397] T 546acbf478ac442b88ff0569bf7ea485 P 23a7821f743042fb9fa18864c1487c4c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 16:02:29.069823 1688 ts_tablet_manager.cc:1397] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 16:02:29.070485 1689 tablet_bootstrap.cc:492] T 546acbf478ac442b88ff0569bf7ea485 P fd528086b31748cfa35d0ec7e5012efd: No bootstrap required, opened a new log
I20260924 16:02:29.070672 1689 ts_tablet_manager.cc:1397] T 546acbf478ac442b88ff0569bf7ea485 P fd528086b31748cfa35d0ec7e5012efd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 16:02:29.072804 1687 raft_consensus.cc:348] T 546acbf478ac442b88ff0569bf7ea485 P 23a7821f743042fb9fa18864c1487c4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } } peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } }
I20260924 16:02:29.073061 1687 raft_consensus.cc:374] T 546acbf478ac442b88ff0569bf7ea485 P 23a7821f743042fb9fa18864c1487c4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:29.073134 1687 raft_consensus.cc:729] T 546acbf478ac442b88ff0569bf7ea485 P 23a7821f743042fb9fa18864c1487c4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23a7821f743042fb9fa18864c1487c4c, State: Initialized, Role: FOLLOWER
I20260924 16:02:29.073589 1687 consensus_queue.cc:260] T 546acbf478ac442b88ff0569bf7ea485 P 23a7821f743042fb9fa18864c1487c4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } } peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } }
I20260924 16:02:29.074239 1659 heartbeater.cc:503] Master 127.25.109.62:46829 was elected leader, sending a full tablet report...
I20260924 16:02:29.074643 1687 ts_tablet_manager.cc:1428] T 546acbf478ac442b88ff0569bf7ea485 P 23a7821f743042fb9fa18864c1487c4c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 16:02:29.074468 1689 raft_consensus.cc:348] T 546acbf478ac442b88ff0569bf7ea485 P fd528086b31748cfa35d0ec7e5012efd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } } peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } }
I20260924 16:02:29.074491 1688 raft_consensus.cc:348] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } } peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } }
I20260924 16:02:29.074717 1689 raft_consensus.cc:374] T 546acbf478ac442b88ff0569bf7ea485 P fd528086b31748cfa35d0ec7e5012efd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:29.074717 1688 raft_consensus.cc:374] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:29.074767 1688 raft_consensus.cc:729] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014fd818c4b34521b5e2587cf075b95a, State: Initialized, Role: FOLLOWER
I20260924 16:02:29.074767 1689 raft_consensus.cc:729] T 546acbf478ac442b88ff0569bf7ea485 P fd528086b31748cfa35d0ec7e5012efd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd528086b31748cfa35d0ec7e5012efd, State: Initialized, Role: FOLLOWER
I20260924 16:02:29.075182 1688 consensus_queue.cc:260] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } } peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } }
I20260924 16:02:29.075182 1689 consensus_queue.cc:260] T 546acbf478ac442b88ff0569bf7ea485 P fd528086b31748cfa35d0ec7e5012efd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } } peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } }
I20260924 16:02:29.075794 1377 heartbeater.cc:503] Master 127.25.109.62:46829 was elected leader, sending a full tablet report...
I20260924 16:02:29.076098 1688 ts_tablet_manager.cc:1428] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 16:02:29.076098 1689 ts_tablet_manager.cc:1428] T 546acbf478ac442b88ff0569bf7ea485 P fd528086b31748cfa35d0ec7e5012efd: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 16:02:29.076395 1517 heartbeater.cc:503] Master 127.25.109.62:46829 was elected leader, sending a full tablet report...
W20260924 16:02:29.234006 1660 tablet.cc:2369] T 546acbf478ac442b88ff0569bf7ea485 P 23a7821f743042fb9fa18864c1487c4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:02:29.264499 1518 tablet.cc:2369] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:02:29.291007 1695 raft_consensus.cc:482] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:02:29.291266 1695 raft_consensus.cc:504] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } } peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } }
I20260924 16:02:29.292603 1695 leader_election.cc:288] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23a7821f743042fb9fa18864c1487c4c (127.25.109.3:35417), fd528086b31748cfa35d0ec7e5012efd (127.25.109.1:36345)
I20260924 16:02:29.296448 1696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:29.292817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:35417 (local address 127.25.109.2:44835)
0924 16:02:29.293135 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:29.293152 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:29.293294 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:29.293585 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:29.293588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:29.293608 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:29.293747 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:29.293752 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.294510 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.294515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:29.295327 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:29.295336 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.295986 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.295990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:29.296085 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:29.296108 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 16:02:29.296203 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 16:02:29.296274 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":104,"threads_started":1}
I20260924 16:02:29.296442 1682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:29.292974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:44835 (local address 127.25.109.3:35417)
0924 16:02:29.293126 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:29.293131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:29.293249 (+ 118us) server_negotiation.cc:407] Connection header received
0924 16:02:29.293417 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:29.293420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:29.293481 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:29.293570 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:29.293852 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.294404 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.295451 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.295852 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.295986 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:29.296041 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:29.296220 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 16:02:29.296288 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 16:02:29.296785 1686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:29.293372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:33391 (local address 127.25.109.1:36345)
0924 16:02:29.293514 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:29.293519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:29.293635 (+ 116us) server_negotiation.cc:407] Connection header received
0924 16:02:29.293787 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:29.293791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:29.293837 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:29.293918 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:29.294205 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.294729 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.295696 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.296200 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.296320 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:29.296370 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:29.296597 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 16:02:29.296671 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 16:02:29.297128 1697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:29.293245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:36345 (local address 127.25.109.2:33391)
0924 16:02:29.293499 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:29.293513 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:29.293683 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:29.293930 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:29.293933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:29.293949 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:29.294076 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:29.294100 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.294843 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.294847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:29.295575 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:29.295584 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.296312 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.296316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:29.296443 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:29.296466 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 16:02:29.296926 (+ 460us) client_negotiation.cc:240] Negotiation successful
0924 16:02:29.297013 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"spinlock_wait_cycles":896,"thread_start_us":91,"threads_started":1}
I20260924 16:02:29.297113 1612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "546acbf478ac442b88ff0569bf7ea485" candidate_uuid: "014fd818c4b34521b5e2587cf075b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a7821f743042fb9fa18864c1487c4c" is_pre_election: true
W20260924 16:02:29.297442 1378 tablet.cc:2369] T 546acbf478ac442b88ff0569bf7ea485 P fd528086b31748cfa35d0ec7e5012efd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:02:29.297456 1612 raft_consensus.cc:2454] T 546acbf478ac442b88ff0569bf7ea485 P 23a7821f743042fb9fa18864c1487c4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 014fd818c4b34521b5e2587cf075b95a in term 0.
I20260924 16:02:29.297842 1330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "546acbf478ac442b88ff0569bf7ea485" candidate_uuid: "014fd818c4b34521b5e2587cf075b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd528086b31748cfa35d0ec7e5012efd" is_pre_election: true
I20260924 16:02:29.297996 1402 leader_election.cc:302] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 014fd818c4b34521b5e2587cf075b95a, 23a7821f743042fb9fa18864c1487c4c; no voters:
I20260924 16:02:29.298115 1330 raft_consensus.cc:2454] T 546acbf478ac442b88ff0569bf7ea485 P fd528086b31748cfa35d0ec7e5012efd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 014fd818c4b34521b5e2587cf075b95a in term 0.
I20260924 16:02:29.298343 1695 raft_consensus.cc:2781] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:02:29.298441 1695 raft_consensus.cc:482] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:02:29.298516 1695 raft_consensus.cc:3037] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:29.299578 1695 raft_consensus.cc:504] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } } peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } }
I20260924 16:02:29.299922 1695 leader_election.cc:288] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a [CANDIDATE]: Term 1 election: Requested vote from peers 23a7821f743042fb9fa18864c1487c4c (127.25.109.3:35417), fd528086b31748cfa35d0ec7e5012efd (127.25.109.1:36345)
I20260924 16:02:29.300282 1612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "546acbf478ac442b88ff0569bf7ea485" candidate_uuid: "014fd818c4b34521b5e2587cf075b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a7821f743042fb9fa18864c1487c4c"
I20260924 16:02:29.300330 1330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "546acbf478ac442b88ff0569bf7ea485" candidate_uuid: "014fd818c4b34521b5e2587cf075b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd528086b31748cfa35d0ec7e5012efd"
I20260924 16:02:29.300454 1330 raft_consensus.cc:3037] T 546acbf478ac442b88ff0569bf7ea485 P fd528086b31748cfa35d0ec7e5012efd [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:29.300436 1612 raft_consensus.cc:3037] T 546acbf478ac442b88ff0569bf7ea485 P 23a7821f743042fb9fa18864c1487c4c [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:29.301842 1330 raft_consensus.cc:2454] T 546acbf478ac442b88ff0569bf7ea485 P fd528086b31748cfa35d0ec7e5012efd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 014fd818c4b34521b5e2587cf075b95a in term 1.
I20260924 16:02:29.302153 1402 leader_election.cc:302] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 014fd818c4b34521b5e2587cf075b95a, fd528086b31748cfa35d0ec7e5012efd; no voters:
I20260924 16:02:29.302269 1612 raft_consensus.cc:2454] T 546acbf478ac442b88ff0569bf7ea485 P 23a7821f743042fb9fa18864c1487c4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 014fd818c4b34521b5e2587cf075b95a in term 1.
I20260924 16:02:29.302369 1695 raft_consensus.cc:2781] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:29.302537 1695 raft_consensus.cc:686] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a [term 1 LEADER]: Becoming Leader. State: Replica: 014fd818c4b34521b5e2587cf075b95a, State: Running, Role: LEADER
I20260924 16:02:29.302881 1695 consensus_queue.cc:237] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } } peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } }
I20260924 16:02:29.306141 1170 catalog_manager.cc:5495] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a reported cstate change: term changed from 0 to 1, leader changed from <none> to 014fd818c4b34521b5e2587cf075b95a (127.25.109.2). New cstate: current_term: 1 leader_uuid: "014fd818c4b34521b5e2587cf075b95a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } health_report { overall_health: HEALTHY } } }
I20260924 16:02:29.348727 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:29.345184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54846 (local address 127.25.109.2:35349)
0924 16:02:29.345308 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:29.345312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:29.345325 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:02:29.345446 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:29.345448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:29.345507 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:29.345600 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:02:29.345811 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.346328 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.347142 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.347347 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.347420 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:29.347653 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:02:29.348111 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:02:29.348208 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:29.348553 (+ 345us) server_negotiation.cc:299] Negotiation successful
0924 16:02:29.348611 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 16:02:29.355048 1330 raft_consensus.cc:1261] T 546acbf478ac442b88ff0569bf7ea485 P fd528086b31748cfa35d0ec7e5012efd [term 1 FOLLOWER]: Refusing update from remote peer 014fd818c4b34521b5e2587cf075b95a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:02:29.355190 1612 raft_consensus.cc:1261] T 546acbf478ac442b88ff0569bf7ea485 P 23a7821f743042fb9fa18864c1487c4c [term 1 FOLLOWER]: Refusing update from remote peer 014fd818c4b34521b5e2587cf075b95a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:02:29.355917 1695 consensus_queue.cc:1035] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:29.356140 1698 consensus_queue.cc:1035] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:29.367210 1703 mvcc.cc:205] Tried to move back new op lower bound from 7332928509350559744 to 7332928509147193344. Current Snapshot: MvccSnapshot[applied={T|T < 7332928509350559744 or (T in {7332928509350559744})}]
I20260924 16:02:29.372001 1170 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:02:29.372200 1170 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:02:29.374704 1170 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c62fa2dda514a4297a2333220ca9ee2: Sending DeleteTablet for 3 replicas of tablet 546acbf478ac442b88ff0569bf7ea485
I20260924 16:02:29.375591 1592 tablet_service.cc:1514] Processing DeleteTablet for tablet 546acbf478ac442b88ff0569bf7ea485 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:29 UTC) from {username='oryx', principal='oryx/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:45466
I20260924 16:02:29.375589 1451 tablet_service.cc:1514] Processing DeleteTablet for tablet 546acbf478ac442b88ff0569bf7ea485 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:29 UTC) from {username='oryx', principal='oryx/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:54844
I20260924 16:02:29.375695 1309 tablet_service.cc:1514] Processing DeleteTablet for tablet 546acbf478ac442b88ff0569bf7ea485 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:29 UTC) from {username='oryx', principal='oryx/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:58766
I20260924 16:02:29.376108 1713 tablet_replica.cc:330] T 546acbf478ac442b88ff0569bf7ea485 P fd528086b31748cfa35d0ec7e5012efd: stopping tablet replica
I20260924 16:02:29.376323 1712 tablet_replica.cc:330] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a: stopping tablet replica
I20260924 16:02:29.376596 1713 raft_consensus.cc:2229] T 546acbf478ac442b88ff0569bf7ea485 P fd528086b31748cfa35d0ec7e5012efd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:29.376637 1712 raft_consensus.cc:2229] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:02:29.376750 1170 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 16:02:29.377012 1712 raft_consensus.cc:2258] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 16:02:29.377202 1170 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:02:29.377230 1713 raft_consensus.cc:2258] T 546acbf478ac442b88ff0569bf7ea485 P fd528086b31748cfa35d0ec7e5012efd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:29.377380 1711 tablet_replica.cc:330] T 546acbf478ac442b88ff0569bf7ea485 P 23a7821f743042fb9fa18864c1487c4c: stopping tablet replica
I20260924 16:02:29.377666 1711 raft_consensus.cc:2229] T 546acbf478ac442b88ff0569bf7ea485 P 23a7821f743042fb9fa18864c1487c4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:29.377964 1711 raft_consensus.cc:2258] T 546acbf478ac442b88ff0569bf7ea485 P 23a7821f743042fb9fa18864c1487c4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:29.378628 1712 ts_tablet_manager.cc:1905] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:29.379710 1713 ts_tablet_manager.cc:1905] T 546acbf478ac442b88ff0569bf7ea485 P fd528086b31748cfa35d0ec7e5012efd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:29.381776 1712 ts_tablet_manager.cc:1918] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:02:29.381875 1712 log.cc:1196] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-1/wal/wals/546acbf478ac442b88ff0569bf7ea485
I20260924 16:02:29.382234 1712 ts_tablet_manager.cc:1939] T 546acbf478ac442b88ff0569bf7ea485 P 014fd818c4b34521b5e2587cf075b95a: Deleting consensus metadata
I20260924 16:02:29.382709 1711 ts_tablet_manager.cc:1905] T 546acbf478ac442b88ff0569bf7ea485 P 23a7821f743042fb9fa18864c1487c4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:29.383332 1309 tablet_service.cc:1467] Processing CreateTablet for tablet d3c7a7094f104bc19345547ed750cac7 (DEFAULT_TABLE table=test-table [id=b2c0556658cd438ca78aaf0d6e43a3b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:29.383630 1309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3c7a7094f104bc19345547ed750cac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:29.383975 1592 tablet_service.cc:1467] Processing CreateTablet for tablet d3c7a7094f104bc19345547ed750cac7 (DEFAULT_TABLE table=test-table [id=b2c0556658cd438ca78aaf0d6e43a3b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:29.384219 1592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3c7a7094f104bc19345547ed750cac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:29.385783 1711 ts_tablet_manager.cc:1918] T 546acbf478ac442b88ff0569bf7ea485 P 23a7821f743042fb9fa18864c1487c4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:02:29.385939 1711 log.cc:1196] T 546acbf478ac442b88ff0569bf7ea485 P 23a7821f743042fb9fa18864c1487c4c: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-2/wal/wals/546acbf478ac442b88ff0569bf7ea485
I20260924 16:02:29.386332 1711 ts_tablet_manager.cc:1939] T 546acbf478ac442b88ff0569bf7ea485 P 23a7821f743042fb9fa18864c1487c4c: Deleting consensus metadata
I20260924 16:02:29.386405 1687 tablet_bootstrap.cc:492] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c: Bootstrap starting.
I20260924 16:02:29.386441 1155 catalog_manager.cc:4841] TS 014fd818c4b34521b5e2587cf075b95a (127.25.109.2:35349): tablet 546acbf478ac442b88ff0569bf7ea485 (table test-table [id=210865ca29074c7e8532d2de13ec50e6]) successfully deleted
I20260924 16:02:29.386653 1451 tablet_service.cc:1467] Processing CreateTablet for tablet d3c7a7094f104bc19345547ed750cac7 (DEFAULT_TABLE table=test-table [id=b2c0556658cd438ca78aaf0d6e43a3b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:29.386909 1451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3c7a7094f104bc19345547ed750cac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:29.386976 1689 tablet_bootstrap.cc:492] T d3c7a7094f104bc19345547ed750cac7 P fd528086b31748cfa35d0ec7e5012efd: Bootstrap starting.
I20260924 16:02:29.387374 1687 tablet_bootstrap.cc:654] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:29.387892 1689 tablet_bootstrap.cc:654] T d3c7a7094f104bc19345547ed750cac7 P fd528086b31748cfa35d0ec7e5012efd: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:29.388648 1687 tablet_bootstrap.cc:492] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c: No bootstrap required, opened a new log
I20260924 16:02:29.388619 1154 catalog_manager.cc:4841] TS 23a7821f743042fb9fa18864c1487c4c (127.25.109.3:35417): tablet 546acbf478ac442b88ff0569bf7ea485 (table test-table [id=210865ca29074c7e8532d2de13ec50e6]) successfully deleted
I20260924 16:02:29.388724 1687 ts_tablet_manager.cc:1397] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 16:02:29.389264 1687 raft_consensus.cc:348] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:29.389369 1687 raft_consensus.cc:374] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:29.389403 1687 raft_consensus.cc:729] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23a7821f743042fb9fa18864c1487c4c, State: Initialized, Role: FOLLOWER
I20260924 16:02:29.389498 1689 tablet_bootstrap.cc:492] T d3c7a7094f104bc19345547ed750cac7 P fd528086b31748cfa35d0ec7e5012efd: No bootstrap required, opened a new log
I20260924 16:02:29.389567 1689 ts_tablet_manager.cc:1397] T d3c7a7094f104bc19345547ed750cac7 P fd528086b31748cfa35d0ec7e5012efd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260924 16:02:29.389531 1687 consensus_queue.cc:260] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:29.389863 1688 tablet_bootstrap.cc:492] T d3c7a7094f104bc19345547ed750cac7 P 014fd818c4b34521b5e2587cf075b95a: Bootstrap starting.
I20260924 16:02:29.390398 1689 raft_consensus.cc:348] T d3c7a7094f104bc19345547ed750cac7 P fd528086b31748cfa35d0ec7e5012efd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:29.390504 1689 raft_consensus.cc:374] T d3c7a7094f104bc19345547ed750cac7 P fd528086b31748cfa35d0ec7e5012efd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:29.390535 1689 raft_consensus.cc:729] T d3c7a7094f104bc19345547ed750cac7 P fd528086b31748cfa35d0ec7e5012efd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd528086b31748cfa35d0ec7e5012efd, State: Initialized, Role: FOLLOWER
I20260924 16:02:29.390652 1689 consensus_queue.cc:260] T d3c7a7094f104bc19345547ed750cac7 P fd528086b31748cfa35d0ec7e5012efd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:29.390820 1688 tablet_bootstrap.cc:654] T d3c7a7094f104bc19345547ed750cac7 P 014fd818c4b34521b5e2587cf075b95a: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:29.390967 1689 ts_tablet_manager.cc:1428] T d3c7a7094f104bc19345547ed750cac7 P fd528086b31748cfa35d0ec7e5012efd: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260924 16:02:29.391151 1713 ts_tablet_manager.cc:1918] T 546acbf478ac442b88ff0569bf7ea485 P fd528086b31748cfa35d0ec7e5012efd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:02:29.391237 1713 log.cc:1196] T 546acbf478ac442b88ff0569bf7ea485 P fd528086b31748cfa35d0ec7e5012efd: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-0/wal/wals/546acbf478ac442b88ff0569bf7ea485
I20260924 16:02:29.391543 1713 ts_tablet_manager.cc:1939] T 546acbf478ac442b88ff0569bf7ea485 P fd528086b31748cfa35d0ec7e5012efd: Deleting consensus metadata
I20260924 16:02:29.392953 1154 catalog_manager.cc:4841] TS fd528086b31748cfa35d0ec7e5012efd (127.25.109.1:36345): tablet 546acbf478ac442b88ff0569bf7ea485 (table test-table [id=210865ca29074c7e8532d2de13ec50e6]) successfully deleted
I20260924 16:02:29.393349 1687 ts_tablet_manager.cc:1428] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260924 16:02:29.394282 1688 tablet_bootstrap.cc:492] T d3c7a7094f104bc19345547ed750cac7 P 014fd818c4b34521b5e2587cf075b95a: No bootstrap required, opened a new log
I20260924 16:02:29.394379 1688 ts_tablet_manager.cc:1397] T d3c7a7094f104bc19345547ed750cac7 P 014fd818c4b34521b5e2587cf075b95a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 16:02:29.394896 1688 raft_consensus.cc:348] T d3c7a7094f104bc19345547ed750cac7 P 014fd818c4b34521b5e2587cf075b95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:29.395040 1688 raft_consensus.cc:374] T d3c7a7094f104bc19345547ed750cac7 P 014fd818c4b34521b5e2587cf075b95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:29.395073 1688 raft_consensus.cc:729] T d3c7a7094f104bc19345547ed750cac7 P 014fd818c4b34521b5e2587cf075b95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014fd818c4b34521b5e2587cf075b95a, State: Initialized, Role: FOLLOWER
I20260924 16:02:29.395200 1688 consensus_queue.cc:260] T d3c7a7094f104bc19345547ed750cac7 P 014fd818c4b34521b5e2587cf075b95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:29.395586 1688 ts_tablet_manager.cc:1428] T d3c7a7094f104bc19345547ed750cac7 P 014fd818c4b34521b5e2587cf075b95a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 16:02:29.634256 1693 raft_consensus.cc:482] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:02:29.634420 1693 raft_consensus.cc:504] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:29.635661 1693 leader_election.cc:288] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd528086b31748cfa35d0ec7e5012efd (127.25.109.1:36345), 014fd818c4b34521b5e2587cf075b95a (127.25.109.2:35349)
I20260924 16:02:29.639655 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:29.636036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:38627 (local address 127.25.109.2:35349)
0924 16:02:29.636187 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:29.636193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:29.636385 (+ 192us) server_negotiation.cc:407] Connection header received
0924 16:02:29.636586 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:29.636589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:29.636635 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:29.636713 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:29.636993 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.637491 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.638639 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.639027 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.639155 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:29.639215 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:29.639441 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 16:02:29.639513 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 16:02:29.639657 1686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:29.636043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:53303 (local address 127.25.109.1:36345)
0924 16:02:29.636182 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:29.636186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:29.636363 (+ 177us) server_negotiation.cc:407] Connection header received
0924 16:02:29.636597 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:29.636600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:29.636649 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:29.636729 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:29.637015 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.637501 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.638647 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.639029 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.639160 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:29.639215 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:29.639448 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 16:02:29.639523 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:02:29.639673 1718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:29.635920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:35349 (local address 127.25.109.3:38627)
0924 16:02:29.636302 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:29.636313 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:29.636458 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:29.636725 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:29.636728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:29.636744 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:29.636888 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:29.636893 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.637602 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.637608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:29.638521 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:29.638531 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.639148 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.639152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:29.639289 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:29.639343 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 16:02:29.639424 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 16:02:29.639509 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":95,"thread_start_us":59,"threads_started":1}
I20260924 16:02:29.639678 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:29.635909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:36345 (local address 127.25.109.3:53303)
0924 16:02:29.636276 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:29.636288 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:29.636477 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:29.636764 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:29.636767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:29.636783 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:29.636908 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:29.636913 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.637609 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.637614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:29.638516 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:29.638526 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.639159 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.639163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:29.639317 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:29.639348 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 16:02:29.639428 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 16:02:29.639506 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":75,"spinlock_wait_cycles":2944,"thread_start_us":74,"threads_started":1}
I20260924 16:02:29.640200 1471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c7a7094f104bc19345547ed750cac7" candidate_uuid: "23a7821f743042fb9fa18864c1487c4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014fd818c4b34521b5e2587cf075b95a" is_pre_election: true
I20260924 16:02:29.640336 1330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c7a7094f104bc19345547ed750cac7" candidate_uuid: "23a7821f743042fb9fa18864c1487c4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd528086b31748cfa35d0ec7e5012efd" is_pre_election: true
I20260924 16:02:29.640506 1471 raft_consensus.cc:2454] T d3c7a7094f104bc19345547ed750cac7 P 014fd818c4b34521b5e2587cf075b95a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23a7821f743042fb9fa18864c1487c4c in term 0.
I20260924 16:02:29.640523 1330 raft_consensus.cc:2454] T d3c7a7094f104bc19345547ed750cac7 P fd528086b31748cfa35d0ec7e5012efd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23a7821f743042fb9fa18864c1487c4c in term 0.
I20260924 16:02:29.640913 1542 leader_election.cc:302] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23a7821f743042fb9fa18864c1487c4c, fd528086b31748cfa35d0ec7e5012efd; no voters:
I20260924 16:02:29.641189 1693 raft_consensus.cc:2781] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:02:29.641263 1693 raft_consensus.cc:482] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:02:29.641301 1693 raft_consensus.cc:3037] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:29.642341 1693 raft_consensus.cc:504] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:29.642702 1693 leader_election.cc:288] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c [CANDIDATE]: Term 1 election: Requested vote from peers fd528086b31748cfa35d0ec7e5012efd (127.25.109.1:36345), 014fd818c4b34521b5e2587cf075b95a (127.25.109.2:35349)
I20260924 16:02:29.643013 1330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c7a7094f104bc19345547ed750cac7" candidate_uuid: "23a7821f743042fb9fa18864c1487c4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd528086b31748cfa35d0ec7e5012efd"
I20260924 16:02:29.643023 1471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c7a7094f104bc19345547ed750cac7" candidate_uuid: "23a7821f743042fb9fa18864c1487c4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014fd818c4b34521b5e2587cf075b95a"
I20260924 16:02:29.643141 1330 raft_consensus.cc:3037] T d3c7a7094f104bc19345547ed750cac7 P fd528086b31748cfa35d0ec7e5012efd [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:29.643150 1471 raft_consensus.cc:3037] T d3c7a7094f104bc19345547ed750cac7 P 014fd818c4b34521b5e2587cf075b95a [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:29.644294 1330 raft_consensus.cc:2454] T d3c7a7094f104bc19345547ed750cac7 P fd528086b31748cfa35d0ec7e5012efd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a7821f743042fb9fa18864c1487c4c in term 1.
I20260924 16:02:29.644292 1471 raft_consensus.cc:2454] T d3c7a7094f104bc19345547ed750cac7 P 014fd818c4b34521b5e2587cf075b95a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a7821f743042fb9fa18864c1487c4c in term 1.
I20260924 16:02:29.644621 1543 leader_election.cc:302] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 014fd818c4b34521b5e2587cf075b95a, 23a7821f743042fb9fa18864c1487c4c, fd528086b31748cfa35d0ec7e5012efd; no voters:
I20260924 16:02:29.644775 1693 raft_consensus.cc:2781] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:29.644935 1693 raft_consensus.cc:686] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c [term 1 LEADER]: Becoming Leader. State: Replica: 23a7821f743042fb9fa18864c1487c4c, State: Running, Role: LEADER
I20260924 16:02:29.645187 1693 consensus_queue.cc:237] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:29.647572 1169 catalog_manager.cc:5495] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 23a7821f743042fb9fa18864c1487c4c (127.25.109.3). New cstate: current_term: 1 leader_uuid: "23a7821f743042fb9fa18864c1487c4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } health_report { overall_health: HEALTHY } } }
Sep 24 16:02:29 dist-test-slave-n106 krb5kdc[1124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265748, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.62@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM
I20260924 16:02:29.710690 1726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:29.702257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:46829 (local address 127.0.0.1:34792)
0924 16:02:29.702616 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:29.702633 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:29.702755 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:29.703023 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:29.703025 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:29.703449 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:29.703556 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:29.703562 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.704285 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.704287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:29.704877 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:29.704885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.705303 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.705305 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:29.705368 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:29.705921 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:29.705932 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:29.707386 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:29.709152 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:29.709162 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:29.709168 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:29.709416 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:29.709696 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:29.709698 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:29.709699 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:29.709784 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:29.710126 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:29.710132 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:29.710248 (+ 116us) client_negotiation.cc:769] Sending connection context
0924 16:02:29.710342 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 16:02:29.710496 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":139,"threads_started":1}
I20260924 16:02:29.710693 1727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:29.702287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34792 (local address 127.25.109.62:46829)
0924 16:02:29.702576 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:29.702583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:29.702716 (+ 133us) server_negotiation.cc:407] Connection header received
0924 16:02:29.702837 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:29.702839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:29.702883 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:29.702968 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:29.703660 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.704192 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.704977 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.705199 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.705277 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:29.707536 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:29.707540 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:29.707542 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:29.707563 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:29.709047 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:29.709529 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:29.709532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:29.709533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:29.709607 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:29.709860 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:29.709862 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:29.709863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:29.710037 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:29.710111 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:29.710394 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 16:02:29.710528 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":215,"thread_start_us":124,"threads_started":1}
I20260924 16:02:29.714972 1169 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:34792:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 16:02:29.715833 1169 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:02:29.720763 1309 tablet_service.cc:1467] Processing CreateTablet for tablet 5a18075de2b34c97bdcfb5fbdfb5b4dc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1edf6e53967c4248b3e1b57c9f70fa84]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:02:29.720878 1451 tablet_service.cc:1467] Processing CreateTablet for tablet 5a18075de2b34c97bdcfb5fbdfb5b4dc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1edf6e53967c4248b3e1b57c9f70fa84]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:02:29.721063 1592 tablet_service.cc:1467] Processing CreateTablet for tablet 5a18075de2b34c97bdcfb5fbdfb5b4dc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1edf6e53967c4248b3e1b57c9f70fa84]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:02:29.721249 1309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a18075de2b34c97bdcfb5fbdfb5b4dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:29.721329 1451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a18075de2b34c97bdcfb5fbdfb5b4dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:29.721451 1592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a18075de2b34c97bdcfb5fbdfb5b4dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:29.723838 1688 tablet_bootstrap.cc:492] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 014fd818c4b34521b5e2587cf075b95a: Bootstrap starting.
I20260924 16:02:29.724357 1687 tablet_bootstrap.cc:492] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 23a7821f743042fb9fa18864c1487c4c: Bootstrap starting.
I20260924 16:02:29.724682 1689 tablet_bootstrap.cc:492] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd: Bootstrap starting.
I20260924 16:02:29.724834 1688 tablet_bootstrap.cc:654] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 014fd818c4b34521b5e2587cf075b95a: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:29.725422 1687 tablet_bootstrap.cc:654] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 23a7821f743042fb9fa18864c1487c4c: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:29.725735 1689 tablet_bootstrap.cc:654] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:29.725939 1688 tablet_bootstrap.cc:492] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 014fd818c4b34521b5e2587cf075b95a: No bootstrap required, opened a new log
I20260924 16:02:29.726035 1688 ts_tablet_manager.cc:1397] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 014fd818c4b34521b5e2587cf075b95a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 16:02:29.726531 1688 raft_consensus.cc:348] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 014fd818c4b34521b5e2587cf075b95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:29.726667 1688 raft_consensus.cc:374] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 014fd818c4b34521b5e2587cf075b95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:29.726708 1688 raft_consensus.cc:729] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 014fd818c4b34521b5e2587cf075b95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014fd818c4b34521b5e2587cf075b95a, State: Initialized, Role: FOLLOWER
I20260924 16:02:29.726846 1688 consensus_queue.cc:260] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 014fd818c4b34521b5e2587cf075b95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:29.726990 1689 tablet_bootstrap.cc:492] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd: No bootstrap required, opened a new log
I20260924 16:02:29.727078 1689 ts_tablet_manager.cc:1397] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 16:02:29.727130 1707 tablet_replica.cc:439] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 014fd818c4b34521b5e2587cf075b95a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } } }
I20260924 16:02:29.727242 1687 tablet_bootstrap.cc:492] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 23a7821f743042fb9fa18864c1487c4c: No bootstrap required, opened a new log
I20260924 16:02:29.727279 1707 tablet_replica.cc:442] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 014fd818c4b34521b5e2587cf075b95a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:02:29.727344 1687 ts_tablet_manager.cc:1397] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 23a7821f743042fb9fa18864c1487c4c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 16:02:29.727533 1689 raft_consensus.cc:348] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:29.727658 1689 raft_consensus.cc:374] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:29.727691 1689 raft_consensus.cc:729] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd528086b31748cfa35d0ec7e5012efd, State: Initialized, Role: FOLLOWER
I20260924 16:02:29.727777 1687 raft_consensus.cc:348] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 23a7821f743042fb9fa18864c1487c4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:29.727808 1689 consensus_queue.cc:260] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:29.727900 1687 raft_consensus.cc:374] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 23a7821f743042fb9fa18864c1487c4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:29.727936 1687 raft_consensus.cc:729] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 23a7821f743042fb9fa18864c1487c4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23a7821f743042fb9fa18864c1487c4c, State: Initialized, Role: FOLLOWER
I20260924 16:02:29.728102 1689 ts_tablet_manager.cc:1428] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 16:02:29.728060 1694 tablet_replica.cc:439] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } } }
I20260924 16:02:29.728134 1687 consensus_queue.cc:260] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 23a7821f743042fb9fa18864c1487c4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:29.728194 1694 tablet_replica.cc:442] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:02:29.728399 1693 tablet_replica.cc:439] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 23a7821f743042fb9fa18864c1487c4c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } } }
I20260924 16:02:29.728478 1687 ts_tablet_manager.cc:1428] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 23a7821f743042fb9fa18864c1487c4c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 16:02:29.728561 1693 tablet_replica.cc:442] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 23a7821f743042fb9fa18864c1487c4c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:02:29.728785 1688 ts_tablet_manager.cc:1428] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 014fd818c4b34521b5e2587cf075b95a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260924 16:02:29.946105 1694 raft_consensus.cc:482] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:02:29.946275 1694 raft_consensus.cc:504] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:29.947367 1694 leader_election.cc:288] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 014fd818c4b34521b5e2587cf075b95a (127.25.109.2:35349), 23a7821f743042fb9fa18864c1487c4c (127.25.109.3:35417)
I20260924 16:02:29.951591 1734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:29.947843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:40367 (local address 127.25.109.3:35417)
0924 16:02:29.948157 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:29.948161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:29.948177 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:02:29.948348 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:29.948351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:29.948395 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:29.948538 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:29.948833 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.949375 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.950527 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.950958 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.951085 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:29.951145 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:29.951359 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 16:02:29.951433 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":73,"threads_started":1}
I20260924 16:02:29.951625 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:29.947843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:36583 (local address 127.25.109.2:35349)
0924 16:02:29.947989 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:29.947996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:29.948245 (+ 249us) server_negotiation.cc:407] Connection header received
0924 16:02:29.948429 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:29.948432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:29.948478 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:29.948574 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:29.948855 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.949374 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.950602 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.951033 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.951188 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:29.951249 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:29.951414 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 16:02:29.951490 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 16:02:29.951656 1732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:29.947696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:35349 (local address 127.25.109.1:36583)
0924 16:02:29.948133 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:29.948146 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:29.948293 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:29.948587 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:29.948590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:29.948630 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:29.948739 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:29.948745 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.949516 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.949519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:29.950452 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:29.950463 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.951162 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.951166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:29.951280 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:29.951305 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 16:02:29.951392 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 16:02:29.951463 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":70,"spinlock_wait_cycles":896,"thread_start_us":69,"threads_started":1}
I20260924 16:02:29.951660 1733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:29.947696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:35417 (local address 127.25.109.1:40367)
0924 16:02:29.948063 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:29.948080 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:29.948202 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:29.948551 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:29.948554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:29.948570 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:29.948722 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:29.948728 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.949492 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.949496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:29.950393 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:29.950404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:29.951083 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:29.951087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:29.951210 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:29.951242 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 16:02:29.951338 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 16:02:29.951414 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":31,"spinlock_wait_cycles":30848,"thread_start_us":98,"threads_started":1}
I20260924 16:02:29.952378 1471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a18075de2b34c97bdcfb5fbdfb5b4dc" candidate_uuid: "fd528086b31748cfa35d0ec7e5012efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014fd818c4b34521b5e2587cf075b95a" is_pre_election: true
I20260924 16:02:29.952495 1612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a18075de2b34c97bdcfb5fbdfb5b4dc" candidate_uuid: "fd528086b31748cfa35d0ec7e5012efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a7821f743042fb9fa18864c1487c4c" is_pre_election: true
I20260924 16:02:29.952596 1471 raft_consensus.cc:2454] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 014fd818c4b34521b5e2587cf075b95a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd528086b31748cfa35d0ec7e5012efd in term 0.
I20260924 16:02:29.952652 1612 raft_consensus.cc:2454] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 23a7821f743042fb9fa18864c1487c4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd528086b31748cfa35d0ec7e5012efd in term 0.
I20260924 16:02:29.952996 1261 leader_election.cc:302] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 014fd818c4b34521b5e2587cf075b95a, fd528086b31748cfa35d0ec7e5012efd; no voters:
I20260924 16:02:29.953277 1694 raft_consensus.cc:2781] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:02:29.953341 1694 raft_consensus.cc:482] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:02:29.953379 1694 raft_consensus.cc:3037] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:29.954391 1694 raft_consensus.cc:504] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:29.954749 1694 leader_election.cc:288] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd [CANDIDATE]: Term 1 election: Requested vote from peers 014fd818c4b34521b5e2587cf075b95a (127.25.109.2:35349), 23a7821f743042fb9fa18864c1487c4c (127.25.109.3:35417)
I20260924 16:02:29.955230 1471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a18075de2b34c97bdcfb5fbdfb5b4dc" candidate_uuid: "fd528086b31748cfa35d0ec7e5012efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014fd818c4b34521b5e2587cf075b95a"
I20260924 16:02:29.955258 1612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a18075de2b34c97bdcfb5fbdfb5b4dc" candidate_uuid: "fd528086b31748cfa35d0ec7e5012efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a7821f743042fb9fa18864c1487c4c"
I20260924 16:02:29.955363 1612 raft_consensus.cc:3037] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 23a7821f743042fb9fa18864c1487c4c [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:29.955363 1471 raft_consensus.cc:3037] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 014fd818c4b34521b5e2587cf075b95a [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:29.956516 1612 raft_consensus.cc:2454] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 23a7821f743042fb9fa18864c1487c4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd528086b31748cfa35d0ec7e5012efd in term 1.
I20260924 16:02:29.956516 1471 raft_consensus.cc:2454] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 014fd818c4b34521b5e2587cf075b95a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd528086b31748cfa35d0ec7e5012efd in term 1.
I20260924 16:02:29.956821 1261 leader_election.cc:302] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 014fd818c4b34521b5e2587cf075b95a, fd528086b31748cfa35d0ec7e5012efd; no voters:
I20260924 16:02:29.957062 1694 raft_consensus.cc:2781] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:29.957229 1694 raft_consensus.cc:686] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd [term 1 LEADER]: Becoming Leader. State: Replica: fd528086b31748cfa35d0ec7e5012efd, State: Running, Role: LEADER
I20260924 16:02:29.957496 1694 consensus_queue.cc:237] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:29.958740 1735 tablet_replica.cc:439] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd: TxnStatusTablet state changed. Reason: New leader fd528086b31748cfa35d0ec7e5012efd. Latest consensus state: current_term: 1 leader_uuid: "fd528086b31748cfa35d0ec7e5012efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } } }
I20260924 16:02:29.958861 1735 tablet_replica.cc:442] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:02:29.959133 1737 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 16:02:29.959293 1737 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 16:02:29.960023 1170 catalog_manager.cc:5495] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd reported cstate change: term changed from 0 to 1, leader changed from <none> to fd528086b31748cfa35d0ec7e5012efd (127.25.109.1). New cstate: current_term: 1 leader_uuid: "fd528086b31748cfa35d0ec7e5012efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } health_report { overall_health: UNKNOWN } } }
I20260924 16:02:30.030108 1726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.025647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:36345 (local address 127.0.0.1:58772)
0924 16:02:30.025859 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:30.025877 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:30.025997 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:30.026382 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:30.026385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:30.026392 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:02:30.026515 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:30.026523 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.027545 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.027548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:30.028169 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:30.028177 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.028806 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.028809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:30.028887 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:30.028942 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:02:30.029809 (+ 867us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:02:30.029826 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 16:02:30.029893 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 16:02:30.029985 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 16:02:30.030120 1686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.025812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58772 (local address 127.25.109.1:36345)
0924 16:02:30.025946 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:30.025951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:30.025968 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:02:30.026153 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:30.026157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:30.026226 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:30.026335 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:02:30.026635 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.027396 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.028294 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.028682 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.028787 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:30.029099 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:02:30.029697 (+ 598us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:02:30.029818 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:30.029908 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 16:02:30.029976 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 16:02:30.033353 1612 raft_consensus.cc:1261] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 23a7821f743042fb9fa18864c1487c4c [term 1 FOLLOWER]: Refusing update from remote peer fd528086b31748cfa35d0ec7e5012efd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:02:30.033630 1471 raft_consensus.cc:1261] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 014fd818c4b34521b5e2587cf075b95a [term 1 FOLLOWER]: Refusing update from remote peer fd528086b31748cfa35d0ec7e5012efd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:02:30.033977 1694 consensus_queue.cc:1035] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd [LEADER]: Connected to new peer: Peer: permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:30.034154 1693 consensus_queue.cc:1035] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:30.034375 1694 consensus_queue.cc:1035] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd [LEADER]: Connected to new peer: Peer: permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:30.035604 1693 tablet_replica.cc:439] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 23a7821f743042fb9fa18864c1487c4c: TxnStatusTablet state changed. Reason: New leader fd528086b31748cfa35d0ec7e5012efd. Latest consensus state: current_term: 1 leader_uuid: "fd528086b31748cfa35d0ec7e5012efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } } }
I20260924 16:02:30.035708 1693 tablet_replica.cc:442] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 23a7821f743042fb9fa18864c1487c4c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:02:30.036355 1707 tablet_replica.cc:439] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 014fd818c4b34521b5e2587cf075b95a: TxnStatusTablet state changed. Reason: New leader fd528086b31748cfa35d0ec7e5012efd. Latest consensus state: current_term: 1 leader_uuid: "fd528086b31748cfa35d0ec7e5012efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } } }
I20260924 16:02:30.036502 1707 tablet_replica.cc:442] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 014fd818c4b34521b5e2587cf075b95a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:02:30.039191 1735 tablet_replica.cc:439] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd528086b31748cfa35d0ec7e5012efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } } }
I20260924 16:02:30.039297 1735 tablet_replica.cc:442] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:02:30.039422 1735 tablet_replica.cc:439] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd528086b31748cfa35d0ec7e5012efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } } }
I20260924 16:02:30.039523 1735 tablet_replica.cc:442] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P fd528086b31748cfa35d0ec7e5012efd: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:02:30.039599 1740 mvcc.cc:205] Tried to move back new op lower bound from 7332928512131411968 to 7332928511828529152. Current Snapshot: MvccSnapshot[applied={T|T < 7332928512131411968}]
I20260924 16:02:30.040213 1707 tablet_replica.cc:439] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 014fd818c4b34521b5e2587cf075b95a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd528086b31748cfa35d0ec7e5012efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } } }
I20260924 16:02:30.040328 1707 tablet_replica.cc:442] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 014fd818c4b34521b5e2587cf075b95a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:02:30.040767 1745 mvcc.cc:205] Tried to move back new op lower bound from 7332928512131411968 to 7332928511828529152. Current Snapshot: MvccSnapshot[applied={T|T < 7332928512131411968}]
I20260924 16:02:30.043354 1693 consensus_queue.cc:1035] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:30.044523 1693 tablet_replica.cc:439] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 23a7821f743042fb9fa18864c1487c4c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd528086b31748cfa35d0ec7e5012efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } } }
I20260924 16:02:30.044628 1693 tablet_replica.cc:442] T 5a18075de2b34c97bdcfb5fbdfb5b4dc P 23a7821f743042fb9fa18864c1487c4c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:02:30.051916 1734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.047618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45474 (local address 127.25.109.3:35417)
0924 16:02:30.047743 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:30.047748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:30.047912 (+ 164us) server_negotiation.cc:407] Connection header received
0924 16:02:30.048047 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:30.048050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:30.048110 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:30.048220 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:02:30.048445 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.049223 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.050122 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.050475 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.050587 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:30.050887 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:02:30.051509 (+ 622us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:02:30.051651 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:30.051716 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 16:02:30.051778 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 16:02:30.063043 1686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.059554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:38519 (local address 127.25.109.1:36345)
0924 16:02:30.059674 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:30.059678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:30.059923 (+ 245us) server_negotiation.cc:407] Connection header received
0924 16:02:30.060093 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:30.060097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:30.060153 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:30.060227 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:02:30.060535 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.061040 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.061917 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.062151 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.062252 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:30.062564 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:02:30.062647 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:02:30.062722 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:30.062845 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 16:02:30.062900 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 16:02:30.063062 1756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.059458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:36345 (local address 127.25.109.3:38519)
0924 16:02:30.059826 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:30.059839 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:30.059962 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:30.060253 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:30.060256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:30.060264 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:02:30.060371 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:30.060376 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.061148 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.061154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:30.061786 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:30.061796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.062292 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.062299 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:30.062380 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:30.062423 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:02:30.062738 (+ 315us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:02:30.062745 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 16:02:30.062820 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 16:02:30.062902 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":112,"threads_started":1}
I20260924 16:02:30.092543 1759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.088563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:35417 (local address 127.25.109.1:60849)
0924 16:02:30.088943 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:30.088957 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:30.089045 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:30.089420 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:30.089424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:30.089433 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:02:30.089573 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:30.089581 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.090409 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.090422 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:30.091250 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:30.091261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.091725 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.091729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:30.091842 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:30.091891 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:02:30.092195 (+ 304us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:02:30.092205 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 16:02:30.092265 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 16:02:30.092355 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":125,"threads_started":1}
I20260924 16:02:30.092936 1734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.088611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:60849 (local address 127.25.109.3:35417)
0924 16:02:30.088752 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:30.088757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:30.089013 (+ 256us) server_negotiation.cc:407] Connection header received
0924 16:02:30.089128 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:30.089131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:30.089212 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:30.089294 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:02:30.089714 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.090286 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.091382 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.091616 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.091724 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:30.092032 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:02:30.092114 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:02:30.092189 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:30.092286 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 16:02:30.092339 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 16:02:30.117460 1168 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:02:30.117648 1168 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:02:30.120157 1168 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c62fa2dda514a4297a2333220ca9ee2: Sending DeleteTablet for 3 replicas of tablet d3c7a7094f104bc19345547ed750cac7
I20260924 16:02:30.120896 1592 tablet_service.cc:1514] Processing DeleteTablet for tablet d3c7a7094f104bc19345547ed750cac7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:30 UTC) from {username='oryx', principal='oryx/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:45466
I20260924 16:02:30.120944 1451 tablet_service.cc:1514] Processing DeleteTablet for tablet d3c7a7094f104bc19345547ed750cac7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:30 UTC) from {username='oryx', principal='oryx/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:54844
I20260924 16:02:30.121616 1309 tablet_service.cc:1514] Processing DeleteTablet for tablet d3c7a7094f104bc19345547ed750cac7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:30 UTC) from {username='oryx', principal='oryx/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:58766
I20260924 16:02:30.121766 1764 tablet_replica.cc:330] T d3c7a7094f104bc19345547ed750cac7 P 014fd818c4b34521b5e2587cf075b95a: stopping tablet replica
I20260924 16:02:30.121898 1764 raft_consensus.cc:2229] T d3c7a7094f104bc19345547ed750cac7 P 014fd818c4b34521b5e2587cf075b95a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:30.122033 1763 tablet_replica.cc:330] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c: stopping tablet replica
I20260924 16:02:30.122175 1764 raft_consensus.cc:2258] T d3c7a7094f104bc19345547ed750cac7 P 014fd818c4b34521b5e2587cf075b95a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:30.122794 1763 raft_consensus.cc:2229] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:02:30.123073 1764 ts_tablet_manager.cc:1905] T d3c7a7094f104bc19345547ed750cac7 P 014fd818c4b34521b5e2587cf075b95a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:30.123190 1763 raft_consensus.cc:2258] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:30.123840 1763 ts_tablet_manager.cc:1905] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:30.124089 1765 tablet_replica.cc:330] T d3c7a7094f104bc19345547ed750cac7 P fd528086b31748cfa35d0ec7e5012efd: stopping tablet replica
I20260924 16:02:30.124243 1765 raft_consensus.cc:2229] T d3c7a7094f104bc19345547ed750cac7 P fd528086b31748cfa35d0ec7e5012efd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:30.124509 1765 raft_consensus.cc:2258] T d3c7a7094f104bc19345547ed750cac7 P fd528086b31748cfa35d0ec7e5012efd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:30.126595 1764 ts_tablet_manager.cc:1918] T d3c7a7094f104bc19345547ed750cac7 P 014fd818c4b34521b5e2587cf075b95a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:02:30.126683 1764 log.cc:1196] T d3c7a7094f104bc19345547ed750cac7 P 014fd818c4b34521b5e2587cf075b95a: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-1/wal/wals/d3c7a7094f104bc19345547ed750cac7
I20260924 16:02:30.127015 1764 ts_tablet_manager.cc:1939] T d3c7a7094f104bc19345547ed750cac7 P 014fd818c4b34521b5e2587cf075b95a: Deleting consensus metadata
I20260924 16:02:30.127513 1765 ts_tablet_manager.cc:1905] T d3c7a7094f104bc19345547ed750cac7 P fd528086b31748cfa35d0ec7e5012efd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:30.127808 1155 catalog_manager.cc:4841] TS 014fd818c4b34521b5e2587cf075b95a (127.25.109.2:35349): tablet d3c7a7094f104bc19345547ed750cac7 (table test-table [id=b2c0556658cd438ca78aaf0d6e43a3b0]) successfully deleted
I20260924 16:02:30.130707 1765 ts_tablet_manager.cc:1918] T d3c7a7094f104bc19345547ed750cac7 P fd528086b31748cfa35d0ec7e5012efd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:02:30.130741 1763 ts_tablet_manager.cc:1918] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:02:30.130828 1763 log.cc:1196] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-2/wal/wals/d3c7a7094f104bc19345547ed750cac7
I20260924 16:02:30.130921 1765 log.cc:1196] T d3c7a7094f104bc19345547ed750cac7 P fd528086b31748cfa35d0ec7e5012efd: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-0/wal/wals/d3c7a7094f104bc19345547ed750cac7
I20260924 16:02:30.131107 1763 ts_tablet_manager.cc:1939] T d3c7a7094f104bc19345547ed750cac7 P 23a7821f743042fb9fa18864c1487c4c: Deleting consensus metadata
I20260924 16:02:30.131291 1765 ts_tablet_manager.cc:1939] T d3c7a7094f104bc19345547ed750cac7 P fd528086b31748cfa35d0ec7e5012efd: Deleting consensus metadata
I20260924 16:02:30.131964 1154 catalog_manager.cc:4841] TS 23a7821f743042fb9fa18864c1487c4c (127.25.109.3:35417): tablet d3c7a7094f104bc19345547ed750cac7 (table test-table [id=b2c0556658cd438ca78aaf0d6e43a3b0]) successfully deleted
I20260924 16:02:30.132197 1154 catalog_manager.cc:4841] TS fd528086b31748cfa35d0ec7e5012efd (127.25.109.1:36345): tablet d3c7a7094f104bc19345547ed750cac7 (table test-table [id=b2c0556658cd438ca78aaf0d6e43a3b0]) successfully deleted
Sep 24 16:02:30 dist-test-slave-n106 krb5kdc[1124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 16:02:30 dist-test-slave-n106 krb5kdc[1124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:30 dist-test-slave-n106 krb5kdc[1124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 16:02:30.155323 1727 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:30.148141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34802 (local address 127.25.109.62:46829)
0924 16:02:30.148310 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:30.148316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:30.148621 (+ 305us) server_negotiation.cc:407] Connection header received
0924 16:02:30.148812 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:30.148815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:30.148861 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:30.148962 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:30.149725 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.150284 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.151276 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.151530 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.151638 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:30.155183 (+ 3545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34802: BlockingRecv error: recv got EOF from 127.0.0.1:34802 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 16:02:30 dist-test-slave-n106 krb5kdc[1124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM
I20260924 16:02:30.166097 1727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.157217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34804 (local address 127.25.109.62:46829)
0924 16:02:30.157351 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:30.157355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:30.157505 (+ 150us) server_negotiation.cc:407] Connection header received
0924 16:02:30.157606 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:30.157609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:30.157658 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:30.157746 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:30.158299 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.159108 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.159914 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.160272 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.160379 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:30.162762 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:30.162766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:30.162769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:30.162799 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:30.164453 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:30.164851 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:30.164855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:30.164857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:30.164951 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:30.165203 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:30.165206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:30.165208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:30.165432 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:30.165576 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:30.165778 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 16:02:30.165913 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 16:02:30.168998 1168 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:02:30.169510 1168 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:02:30.174582 1309 tablet_service.cc:1467] Processing CreateTablet for tablet 48e44faf99244e668216bb23b0a44caa (DEFAULT_TABLE table=test-table [id=d641715acde443898c21ea450507fb6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:30.174659 1451 tablet_service.cc:1467] Processing CreateTablet for tablet 48e44faf99244e668216bb23b0a44caa (DEFAULT_TABLE table=test-table [id=d641715acde443898c21ea450507fb6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:30.174907 1309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e44faf99244e668216bb23b0a44caa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:30.175015 1451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e44faf99244e668216bb23b0a44caa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:30.176748 1592 tablet_service.cc:1467] Processing CreateTablet for tablet 48e44faf99244e668216bb23b0a44caa (DEFAULT_TABLE table=test-table [id=d641715acde443898c21ea450507fb6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:30.177044 1592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e44faf99244e668216bb23b0a44caa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:30.177289 1688 tablet_bootstrap.cc:492] T 48e44faf99244e668216bb23b0a44caa P 014fd818c4b34521b5e2587cf075b95a: Bootstrap starting.
I20260924 16:02:30.178215 1688 tablet_bootstrap.cc:654] T 48e44faf99244e668216bb23b0a44caa P 014fd818c4b34521b5e2587cf075b95a: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:30.178814 1689 tablet_bootstrap.cc:492] T 48e44faf99244e668216bb23b0a44caa P fd528086b31748cfa35d0ec7e5012efd: Bootstrap starting.
I20260924 16:02:30.179574 1687 tablet_bootstrap.cc:492] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c: Bootstrap starting.
I20260924 16:02:30.179790 1689 tablet_bootstrap.cc:654] T 48e44faf99244e668216bb23b0a44caa P fd528086b31748cfa35d0ec7e5012efd: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:30.180711 1687 tablet_bootstrap.cc:654] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:30.181293 1689 tablet_bootstrap.cc:492] T 48e44faf99244e668216bb23b0a44caa P fd528086b31748cfa35d0ec7e5012efd: No bootstrap required, opened a new log
I20260924 16:02:30.181406 1689 ts_tablet_manager.cc:1397] T 48e44faf99244e668216bb23b0a44caa P fd528086b31748cfa35d0ec7e5012efd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260924 16:02:30.181583 1688 tablet_bootstrap.cc:492] T 48e44faf99244e668216bb23b0a44caa P 014fd818c4b34521b5e2587cf075b95a: No bootstrap required, opened a new log
I20260924 16:02:30.181659 1688 ts_tablet_manager.cc:1397] T 48e44faf99244e668216bb23b0a44caa P 014fd818c4b34521b5e2587cf075b95a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 16:02:30.181879 1689 raft_consensus.cc:348] T 48e44faf99244e668216bb23b0a44caa P fd528086b31748cfa35d0ec7e5012efd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:30.181993 1689 raft_consensus.cc:374] T 48e44faf99244e668216bb23b0a44caa P fd528086b31748cfa35d0ec7e5012efd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:30.182027 1689 raft_consensus.cc:729] T 48e44faf99244e668216bb23b0a44caa P fd528086b31748cfa35d0ec7e5012efd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd528086b31748cfa35d0ec7e5012efd, State: Initialized, Role: FOLLOWER
I20260924 16:02:30.182045 1688 raft_consensus.cc:348] T 48e44faf99244e668216bb23b0a44caa P 014fd818c4b34521b5e2587cf075b95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:30.182147 1688 raft_consensus.cc:374] T 48e44faf99244e668216bb23b0a44caa P 014fd818c4b34521b5e2587cf075b95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:30.182180 1688 raft_consensus.cc:729] T 48e44faf99244e668216bb23b0a44caa P 014fd818c4b34521b5e2587cf075b95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014fd818c4b34521b5e2587cf075b95a, State: Initialized, Role: FOLLOWER
I20260924 16:02:30.182165 1689 consensus_queue.cc:260] T 48e44faf99244e668216bb23b0a44caa P fd528086b31748cfa35d0ec7e5012efd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:30.182291 1688 consensus_queue.cc:260] T 48e44faf99244e668216bb23b0a44caa P 014fd818c4b34521b5e2587cf075b95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:30.182464 1687 tablet_bootstrap.cc:492] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c: No bootstrap required, opened a new log
I20260924 16:02:30.182438 1689 ts_tablet_manager.cc:1428] T 48e44faf99244e668216bb23b0a44caa P fd528086b31748cfa35d0ec7e5012efd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:02:30.182534 1688 ts_tablet_manager.cc:1428] T 48e44faf99244e668216bb23b0a44caa P 014fd818c4b34521b5e2587cf075b95a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:02:30.182538 1687 ts_tablet_manager.cc:1397] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260924 16:02:30.182956 1687 raft_consensus.cc:348] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:30.183055 1687 raft_consensus.cc:374] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:30.183084 1687 raft_consensus.cc:729] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23a7821f743042fb9fa18864c1487c4c, State: Initialized, Role: FOLLOWER
I20260924 16:02:30.183195 1687 consensus_queue.cc:260] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:30.183483 1687 ts_tablet_manager.cc:1428] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 16:02:30.390409 1757 raft_consensus.cc:482] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:02:30.390570 1757 raft_consensus.cc:504] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:30.391083 1757 leader_election.cc:288] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd528086b31748cfa35d0ec7e5012efd (127.25.109.1:36345), 014fd818c4b34521b5e2587cf075b95a (127.25.109.2:35349)
I20260924 16:02:30.391662 1471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e44faf99244e668216bb23b0a44caa" candidate_uuid: "23a7821f743042fb9fa18864c1487c4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014fd818c4b34521b5e2587cf075b95a" is_pre_election: true
I20260924 16:02:30.391703 1330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e44faf99244e668216bb23b0a44caa" candidate_uuid: "23a7821f743042fb9fa18864c1487c4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd528086b31748cfa35d0ec7e5012efd" is_pre_election: true
I20260924 16:02:30.391850 1471 raft_consensus.cc:2454] T 48e44faf99244e668216bb23b0a44caa P 014fd818c4b34521b5e2587cf075b95a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23a7821f743042fb9fa18864c1487c4c in term 0.
I20260924 16:02:30.391975 1330 raft_consensus.cc:2454] T 48e44faf99244e668216bb23b0a44caa P fd528086b31748cfa35d0ec7e5012efd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23a7821f743042fb9fa18864c1487c4c in term 0.
I20260924 16:02:30.392217 1543 leader_election.cc:302] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 014fd818c4b34521b5e2587cf075b95a, 23a7821f743042fb9fa18864c1487c4c; no voters:
I20260924 16:02:30.392468 1757 raft_consensus.cc:2781] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:02:30.392534 1757 raft_consensus.cc:482] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:02:30.392580 1757 raft_consensus.cc:3037] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:30.393666 1757 raft_consensus.cc:504] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:30.394019 1757 leader_election.cc:288] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c [CANDIDATE]: Term 1 election: Requested vote from peers fd528086b31748cfa35d0ec7e5012efd (127.25.109.1:36345), 014fd818c4b34521b5e2587cf075b95a (127.25.109.2:35349)
I20260924 16:02:30.394423 1471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e44faf99244e668216bb23b0a44caa" candidate_uuid: "23a7821f743042fb9fa18864c1487c4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014fd818c4b34521b5e2587cf075b95a"
I20260924 16:02:30.394419 1330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e44faf99244e668216bb23b0a44caa" candidate_uuid: "23a7821f743042fb9fa18864c1487c4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd528086b31748cfa35d0ec7e5012efd"
I20260924 16:02:30.394527 1471 raft_consensus.cc:3037] T 48e44faf99244e668216bb23b0a44caa P 014fd818c4b34521b5e2587cf075b95a [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:30.394562 1330 raft_consensus.cc:3037] T 48e44faf99244e668216bb23b0a44caa P fd528086b31748cfa35d0ec7e5012efd [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:30.395499 1330 raft_consensus.cc:2454] T 48e44faf99244e668216bb23b0a44caa P fd528086b31748cfa35d0ec7e5012efd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a7821f743042fb9fa18864c1487c4c in term 1.
I20260924 16:02:30.395547 1471 raft_consensus.cc:2454] T 48e44faf99244e668216bb23b0a44caa P 014fd818c4b34521b5e2587cf075b95a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a7821f743042fb9fa18864c1487c4c in term 1.
I20260924 16:02:30.395815 1542 leader_election.cc:302] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23a7821f743042fb9fa18864c1487c4c, fd528086b31748cfa35d0ec7e5012efd; no voters:
I20260924 16:02:30.395996 1757 raft_consensus.cc:2781] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:30.396097 1757 raft_consensus.cc:686] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c [term 1 LEADER]: Becoming Leader. State: Replica: 23a7821f743042fb9fa18864c1487c4c, State: Running, Role: LEADER
I20260924 16:02:30.396250 1757 consensus_queue.cc:237] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:30.397948 1169 catalog_manager.cc:5495] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 23a7821f743042fb9fa18864c1487c4c (127.25.109.3). New cstate: current_term: 1 leader_uuid: "23a7821f743042fb9fa18864c1487c4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } health_report { overall_health: HEALTHY } } }
I20260924 16:02:30.424458 1734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.419847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45490 (local address 127.25.109.3:35417)
0924 16:02:30.419970 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:30.419973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:30.419986 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:02:30.420140 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:30.420144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:30.420208 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:30.420298 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:02:30.420630 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.421570 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.422950 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.423403 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.423554 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:30.423892 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:02:30.424006 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:02:30.424090 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:30.424211 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 16:02:30.424271 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 16:02:30.426744 1471 raft_consensus.cc:1261] T 48e44faf99244e668216bb23b0a44caa P 014fd818c4b34521b5e2587cf075b95a [term 1 FOLLOWER]: Refusing update from remote peer 23a7821f743042fb9fa18864c1487c4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:02:30.426836 1330 raft_consensus.cc:1261] T 48e44faf99244e668216bb23b0a44caa P fd528086b31748cfa35d0ec7e5012efd [term 1 FOLLOWER]: Refusing update from remote peer 23a7821f743042fb9fa18864c1487c4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:02:30.427330 1757 consensus_queue.cc:1035] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:30.427505 1719 consensus_queue.cc:1035] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:30.435415 1170 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:02:30.435570 1170 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:02:30.437513 1170 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c62fa2dda514a4297a2333220ca9ee2: Sending DeleteTablet for 3 replicas of tablet 48e44faf99244e668216bb23b0a44caa
I20260924 16:02:30.438083 1451 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e44faf99244e668216bb23b0a44caa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:30 UTC) from {username='oryx', principal='oryx/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:54844
I20260924 16:02:30.438179 1592 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e44faf99244e668216bb23b0a44caa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:30 UTC) from {username='oryx', principal='oryx/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:45466
I20260924 16:02:30.438196 1309 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e44faf99244e668216bb23b0a44caa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:30 UTC) from {username='oryx', principal='oryx/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:58766
I20260924 16:02:30.438377 1764 tablet_replica.cc:330] T 48e44faf99244e668216bb23b0a44caa P 014fd818c4b34521b5e2587cf075b95a: stopping tablet replica
I20260924 16:02:30.438380 1765 tablet_replica.cc:330] T 48e44faf99244e668216bb23b0a44caa P fd528086b31748cfa35d0ec7e5012efd: stopping tablet replica
I20260924 16:02:30.438501 1764 raft_consensus.cc:2229] T 48e44faf99244e668216bb23b0a44caa P 014fd818c4b34521b5e2587cf075b95a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:30.438519 1765 raft_consensus.cc:2229] T 48e44faf99244e668216bb23b0a44caa P fd528086b31748cfa35d0ec7e5012efd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:30.438640 1764 raft_consensus.cc:2258] T 48e44faf99244e668216bb23b0a44caa P 014fd818c4b34521b5e2587cf075b95a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:30.438663 1765 raft_consensus.cc:2258] T 48e44faf99244e668216bb23b0a44caa P fd528086b31748cfa35d0ec7e5012efd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:30.438843 1763 tablet_replica.cc:330] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c: stopping tablet replica
I20260924 16:02:30.438978 1763 raft_consensus.cc:2229] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:02:30.439170 1763 raft_consensus.cc:2258] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:30.439545 1765 ts_tablet_manager.cc:1905] T 48e44faf99244e668216bb23b0a44caa P fd528086b31748cfa35d0ec7e5012efd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:30.439594 1764 ts_tablet_manager.cc:1905] T 48e44faf99244e668216bb23b0a44caa P 014fd818c4b34521b5e2587cf075b95a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:30.439545 1170 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 16:02:30.439832 1763 ts_tablet_manager.cc:1905] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 16:02:30.439950 1170 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:02:30.442062 1764 ts_tablet_manager.cc:1918] T 48e44faf99244e668216bb23b0a44caa P 014fd818c4b34521b5e2587cf075b95a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:02:30.442130 1764 log.cc:1196] T 48e44faf99244e668216bb23b0a44caa P 014fd818c4b34521b5e2587cf075b95a: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-1/wal/wals/48e44faf99244e668216bb23b0a44caa
I20260924 16:02:30.442199 1763 ts_tablet_manager.cc:1918] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:02:30.442199 1765 ts_tablet_manager.cc:1918] T 48e44faf99244e668216bb23b0a44caa P fd528086b31748cfa35d0ec7e5012efd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:02:30.442269 1763 log.cc:1196] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-2/wal/wals/48e44faf99244e668216bb23b0a44caa
I20260924 16:02:30.442269 1765 log.cc:1196] T 48e44faf99244e668216bb23b0a44caa P fd528086b31748cfa35d0ec7e5012efd: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-0/wal/wals/48e44faf99244e668216bb23b0a44caa
I20260924 16:02:30.442406 1764 ts_tablet_manager.cc:1939] T 48e44faf99244e668216bb23b0a44caa P 014fd818c4b34521b5e2587cf075b95a: Deleting consensus metadata
I20260924 16:02:30.442534 1763 ts_tablet_manager.cc:1939] T 48e44faf99244e668216bb23b0a44caa P 23a7821f743042fb9fa18864c1487c4c: Deleting consensus metadata
I20260924 16:02:30.442536 1765 ts_tablet_manager.cc:1939] T 48e44faf99244e668216bb23b0a44caa P fd528086b31748cfa35d0ec7e5012efd: Deleting consensus metadata
I20260924 16:02:30.443177 1154 catalog_manager.cc:4841] TS fd528086b31748cfa35d0ec7e5012efd (127.25.109.1:36345): tablet 48e44faf99244e668216bb23b0a44caa (table test-table [id=d641715acde443898c21ea450507fb6e]) successfully deleted
I20260924 16:02:30.443336 1154 catalog_manager.cc:4841] TS 23a7821f743042fb9fa18864c1487c4c (127.25.109.3:35417): tablet 48e44faf99244e668216bb23b0a44caa (table test-table [id=d641715acde443898c21ea450507fb6e]) successfully deleted
I20260924 16:02:30.443747 1155 catalog_manager.cc:4841] TS 014fd818c4b34521b5e2587cf075b95a (127.25.109.2:35349): tablet 48e44faf99244e668216bb23b0a44caa (table test-table [id=d641715acde443898c21ea450507fb6e]) successfully deleted
I20260924 16:02:30.447798 1451 tablet_service.cc:1467] Processing CreateTablet for tablet ac17ef8e8dff4178bec0c736ae9a42b2 (DEFAULT_TABLE table=test-table [id=e7fed9996f6e4d5695a5b17e1180181b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:30.447798 1309 tablet_service.cc:1467] Processing CreateTablet for tablet ac17ef8e8dff4178bec0c736ae9a42b2 (DEFAULT_TABLE table=test-table [id=e7fed9996f6e4d5695a5b17e1180181b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:30.447887 1592 tablet_service.cc:1467] Processing CreateTablet for tablet ac17ef8e8dff4178bec0c736ae9a42b2 (DEFAULT_TABLE table=test-table [id=e7fed9996f6e4d5695a5b17e1180181b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:30.448089 1309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac17ef8e8dff4178bec0c736ae9a42b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:30.448107 1451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac17ef8e8dff4178bec0c736ae9a42b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:30.448146 1592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac17ef8e8dff4178bec0c736ae9a42b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:30.450083 1689 tablet_bootstrap.cc:492] T ac17ef8e8dff4178bec0c736ae9a42b2 P fd528086b31748cfa35d0ec7e5012efd: Bootstrap starting.
I20260924 16:02:30.450407 1688 tablet_bootstrap.cc:492] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a: Bootstrap starting.
I20260924 16:02:30.450407 1687 tablet_bootstrap.cc:492] T ac17ef8e8dff4178bec0c736ae9a42b2 P 23a7821f743042fb9fa18864c1487c4c: Bootstrap starting.
I20260924 16:02:30.451078 1689 tablet_bootstrap.cc:654] T ac17ef8e8dff4178bec0c736ae9a42b2 P fd528086b31748cfa35d0ec7e5012efd: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:30.451188 1688 tablet_bootstrap.cc:654] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:30.451187 1687 tablet_bootstrap.cc:654] T ac17ef8e8dff4178bec0c736ae9a42b2 P 23a7821f743042fb9fa18864c1487c4c: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:30.452389 1689 tablet_bootstrap.cc:492] T ac17ef8e8dff4178bec0c736ae9a42b2 P fd528086b31748cfa35d0ec7e5012efd: No bootstrap required, opened a new log
I20260924 16:02:30.452464 1687 tablet_bootstrap.cc:492] T ac17ef8e8dff4178bec0c736ae9a42b2 P 23a7821f743042fb9fa18864c1487c4c: No bootstrap required, opened a new log
I20260924 16:02:30.452500 1689 ts_tablet_manager.cc:1397] T ac17ef8e8dff4178bec0c736ae9a42b2 P fd528086b31748cfa35d0ec7e5012efd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 16:02:30.452543 1687 ts_tablet_manager.cc:1397] T ac17ef8e8dff4178bec0c736ae9a42b2 P 23a7821f743042fb9fa18864c1487c4c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260924 16:02:30.452725 1688 tablet_bootstrap.cc:492] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a: No bootstrap required, opened a new log
I20260924 16:02:30.452783 1688 ts_tablet_manager.cc:1397] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 16:02:30.452910 1687 raft_consensus.cc:348] T ac17ef8e8dff4178bec0c736ae9a42b2 P 23a7821f743042fb9fa18864c1487c4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:30.452910 1689 raft_consensus.cc:348] T ac17ef8e8dff4178bec0c736ae9a42b2 P fd528086b31748cfa35d0ec7e5012efd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:30.453027 1689 raft_consensus.cc:374] T ac17ef8e8dff4178bec0c736ae9a42b2 P fd528086b31748cfa35d0ec7e5012efd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:30.453037 1687 raft_consensus.cc:374] T ac17ef8e8dff4178bec0c736ae9a42b2 P 23a7821f743042fb9fa18864c1487c4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:30.453066 1689 raft_consensus.cc:729] T ac17ef8e8dff4178bec0c736ae9a42b2 P fd528086b31748cfa35d0ec7e5012efd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd528086b31748cfa35d0ec7e5012efd, State: Initialized, Role: FOLLOWER
I20260924 16:02:30.453073 1687 raft_consensus.cc:729] T ac17ef8e8dff4178bec0c736ae9a42b2 P 23a7821f743042fb9fa18864c1487c4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23a7821f743042fb9fa18864c1487c4c, State: Initialized, Role: FOLLOWER
I20260924 16:02:30.453078 1688 raft_consensus.cc:348] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:30.453186 1688 raft_consensus.cc:374] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:30.453222 1688 raft_consensus.cc:729] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014fd818c4b34521b5e2587cf075b95a, State: Initialized, Role: FOLLOWER
I20260924 16:02:30.453192 1687 consensus_queue.cc:260] T ac17ef8e8dff4178bec0c736ae9a42b2 P 23a7821f743042fb9fa18864c1487c4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:30.453192 1689 consensus_queue.cc:260] T ac17ef8e8dff4178bec0c736ae9a42b2 P fd528086b31748cfa35d0ec7e5012efd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:30.453357 1688 consensus_queue.cc:260] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:30.453447 1687 ts_tablet_manager.cc:1428] T ac17ef8e8dff4178bec0c736ae9a42b2 P 23a7821f743042fb9fa18864c1487c4c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:02:30.453452 1689 ts_tablet_manager.cc:1428] T ac17ef8e8dff4178bec0c736ae9a42b2 P fd528086b31748cfa35d0ec7e5012efd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:02:30.453630 1688 ts_tablet_manager.cc:1428] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 16:02:30.589890 1751 raft_consensus.cc:482] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:02:30.590059 1751 raft_consensus.cc:504] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:30.590530 1751 leader_election.cc:288] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd528086b31748cfa35d0ec7e5012efd (127.25.109.1:36345), 23a7821f743042fb9fa18864c1487c4c (127.25.109.3:35417)
I20260924 16:02:30.591081 1330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac17ef8e8dff4178bec0c736ae9a42b2" candidate_uuid: "014fd818c4b34521b5e2587cf075b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd528086b31748cfa35d0ec7e5012efd" is_pre_election: true
I20260924 16:02:30.591142 1612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac17ef8e8dff4178bec0c736ae9a42b2" candidate_uuid: "014fd818c4b34521b5e2587cf075b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a7821f743042fb9fa18864c1487c4c" is_pre_election: true
I20260924 16:02:30.591249 1330 raft_consensus.cc:2454] T ac17ef8e8dff4178bec0c736ae9a42b2 P fd528086b31748cfa35d0ec7e5012efd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 014fd818c4b34521b5e2587cf075b95a in term 0.
I20260924 16:02:30.591341 1612 raft_consensus.cc:2454] T ac17ef8e8dff4178bec0c736ae9a42b2 P 23a7821f743042fb9fa18864c1487c4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 014fd818c4b34521b5e2587cf075b95a in term 0.
I20260924 16:02:30.591627 1402 leader_election.cc:302] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 014fd818c4b34521b5e2587cf075b95a, fd528086b31748cfa35d0ec7e5012efd; no voters:
I20260924 16:02:30.591848 1751 raft_consensus.cc:2781] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:02:30.591918 1751 raft_consensus.cc:482] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:02:30.591970 1751 raft_consensus.cc:3037] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:30.593113 1751 raft_consensus.cc:504] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:30.593430 1751 leader_election.cc:288] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a [CANDIDATE]: Term 1 election: Requested vote from peers fd528086b31748cfa35d0ec7e5012efd (127.25.109.1:36345), 23a7821f743042fb9fa18864c1487c4c (127.25.109.3:35417)
I20260924 16:02:30.593869 1612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac17ef8e8dff4178bec0c736ae9a42b2" candidate_uuid: "014fd818c4b34521b5e2587cf075b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a7821f743042fb9fa18864c1487c4c"
I20260924 16:02:30.593868 1330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac17ef8e8dff4178bec0c736ae9a42b2" candidate_uuid: "014fd818c4b34521b5e2587cf075b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd528086b31748cfa35d0ec7e5012efd"
I20260924 16:02:30.593987 1612 raft_consensus.cc:3037] T ac17ef8e8dff4178bec0c736ae9a42b2 P 23a7821f743042fb9fa18864c1487c4c [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:30.594000 1330 raft_consensus.cc:3037] T ac17ef8e8dff4178bec0c736ae9a42b2 P fd528086b31748cfa35d0ec7e5012efd [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:30.594890 1612 raft_consensus.cc:2454] T ac17ef8e8dff4178bec0c736ae9a42b2 P 23a7821f743042fb9fa18864c1487c4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 014fd818c4b34521b5e2587cf075b95a in term 1.
I20260924 16:02:30.594959 1330 raft_consensus.cc:2454] T ac17ef8e8dff4178bec0c736ae9a42b2 P fd528086b31748cfa35d0ec7e5012efd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 014fd818c4b34521b5e2587cf075b95a in term 1.
I20260924 16:02:30.595221 1402 leader_election.cc:302] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 014fd818c4b34521b5e2587cf075b95a, 23a7821f743042fb9fa18864c1487c4c; no voters:
I20260924 16:02:30.595414 1751 raft_consensus.cc:2781] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:30.595602 1751 raft_consensus.cc:686] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a [term 1 LEADER]: Becoming Leader. State: Replica: 014fd818c4b34521b5e2587cf075b95a, State: Running, Role: LEADER
I20260924 16:02:30.595762 1751 consensus_queue.cc:237] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } }
I20260924 16:02:30.597354 1169 catalog_manager.cc:5495] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a reported cstate change: term changed from 0 to 1, leader changed from <none> to 014fd818c4b34521b5e2587cf075b95a (127.25.109.2). New cstate: current_term: 1 leader_uuid: "014fd818c4b34521b5e2587cf075b95a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "014fd818c4b34521b5e2587cf075b95a" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 35349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 } health_report { overall_health: UNKNOWN } } }
I20260924 16:02:30.612071 1791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.608989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54850 (local address 127.25.109.2:35349)
0924 16:02:30.609219 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:30.609222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:30.609233 (+ 11us) server_negotiation.cc:407] Connection header received
0924 16:02:30.609270 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:30.609272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:30.609311 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:30.609379 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:02:30.609581 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.610180 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.611087 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.611303 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.611373 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:30.611650 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:02:30.611716 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:02:30.611767 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:30.611859 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 16:02:30.611901 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":106,"threads_started":1}
I20260924 16:02:30.621649 1794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.618414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:36345 (local address 127.25.109.2:48865)
0924 16:02:30.618710 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:30.618730 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:30.618805 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:30.619007 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:30.619009 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:30.619031 (+ 22us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:02:30.619111 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:30.619115 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.619873 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.619875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:30.620590 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:30.620599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.621031 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.621033 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:30.621091 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:30.621143 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:02:30.621415 (+ 272us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:02:30.621420 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 16:02:30.621468 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 16:02:30.621530 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":93,"threads_started":1}
I20260924 16:02:30.621774 1795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.618525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:48865 (local address 127.25.109.1:36345)
0924 16:02:30.618786 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:30.618789 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:30.618802 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:02:30.618873 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:30.618875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:30.618920 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:30.618992 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:02:30.619267 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.619780 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.620699 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.620935 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.621027 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:30.621272 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:02:30.621349 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:02:30.621403 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:30.621475 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 16:02:30.621566 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":102,"threads_started":1}
I20260924 16:02:30.627651 1612 raft_consensus.cc:1261] T ac17ef8e8dff4178bec0c736ae9a42b2 P 23a7821f743042fb9fa18864c1487c4c [term 1 FOLLOWER]: Refusing update from remote peer 014fd818c4b34521b5e2587cf075b95a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:02:30.627671 1330 raft_consensus.cc:1261] T ac17ef8e8dff4178bec0c736ae9a42b2 P fd528086b31748cfa35d0ec7e5012efd [term 1 FOLLOWER]: Refusing update from remote peer 014fd818c4b34521b5e2587cf075b95a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:02:30.628058 1789 consensus_queue.cc:1035] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "23a7821f743042fb9fa18864c1487c4c" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:30.628261 1751 consensus_queue.cc:1035] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd528086b31748cfa35d0ec7e5012efd" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:30.644712 1791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.641617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:52435 (local address 127.25.109.2:35349)
0924 16:02:30.641730 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:30.641734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:30.641761 (+ 27us) server_negotiation.cc:407] Connection header received
0924 16:02:30.641798 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:30.641800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:30.641839 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:30.641894 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:02:30.642085 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.642577 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.643468 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.643808 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.643939 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:30.644200 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:02:30.644293 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:02:30.644366 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:30.644529 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 16:02:30.644595 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 16:02:30.644770 1799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.641304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:35349 (local address 127.25.109.1:52435)
0924 16:02:30.641578 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:30.641596 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:30.641688 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:30.641912 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:30.641914 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:30.641921 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:02:30.642020 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:30.642024 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.642658 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.642660 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:30.643307 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:30.643318 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:30.643942 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:30.643946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:30.644049 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:30.644084 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:02:30.644369 (+ 285us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:02:30.644375 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 16:02:30.644487 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 16:02:30.644599 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":95,"threads_started":1}
I20260924 16:02:30.664750 1168 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:02:30.664908 1168 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:02:30.666891 1168 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c62fa2dda514a4297a2333220ca9ee2: Sending DeleteTablet for 3 replicas of tablet ac17ef8e8dff4178bec0c736ae9a42b2
I20260924 16:02:30.667387 1309 tablet_service.cc:1514] Processing DeleteTablet for tablet ac17ef8e8dff4178bec0c736ae9a42b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:30 UTC) from {username='oryx', principal='oryx/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:58766
I20260924 16:02:30.667387 1451 tablet_service.cc:1514] Processing DeleteTablet for tablet ac17ef8e8dff4178bec0c736ae9a42b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:30 UTC) from {username='oryx', principal='oryx/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:54844
I20260924 16:02:30.667472 1592 tablet_service.cc:1514] Processing DeleteTablet for tablet ac17ef8e8dff4178bec0c736ae9a42b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:30 UTC) from {username='oryx', principal='oryx/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:45466
I20260924 16:02:30.667567 1765 tablet_replica.cc:330] T ac17ef8e8dff4178bec0c736ae9a42b2 P fd528086b31748cfa35d0ec7e5012efd: stopping tablet replica
I20260924 16:02:30.667689 1763 tablet_replica.cc:330] T ac17ef8e8dff4178bec0c736ae9a42b2 P 23a7821f743042fb9fa18864c1487c4c: stopping tablet replica
I20260924 16:02:30.667691 1764 tablet_replica.cc:330] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a: stopping tablet replica
I20260924 16:02:30.667706 1765 raft_consensus.cc:2229] T ac17ef8e8dff4178bec0c736ae9a42b2 P fd528086b31748cfa35d0ec7e5012efd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:30.667798 1763 raft_consensus.cc:2229] T ac17ef8e8dff4178bec0c736ae9a42b2 P 23a7821f743042fb9fa18864c1487c4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:30.667817 1765 raft_consensus.cc:2258] T ac17ef8e8dff4178bec0c736ae9a42b2 P fd528086b31748cfa35d0ec7e5012efd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:30.667814 1764 raft_consensus.cc:2229] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:02:30.667901 1763 raft_consensus.cc:2258] T ac17ef8e8dff4178bec0c736ae9a42b2 P 23a7821f743042fb9fa18864c1487c4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:30.668000 1764 raft_consensus.cc:2258] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:30.668182 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 1244
I20260924 16:02:30.670585 1764 ts_tablet_manager.cc:1905] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:30.670611 1763 ts_tablet_manager.cc:1905] T ac17ef8e8dff4178bec0c736ae9a42b2 P 23a7821f743042fb9fa18864c1487c4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:30.673174 1763 ts_tablet_manager.cc:1918] T ac17ef8e8dff4178bec0c736ae9a42b2 P 23a7821f743042fb9fa18864c1487c4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:02:30.673174 1764 ts_tablet_manager.cc:1918] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:02:30.673259 1764 log.cc:1196] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-1/wal/wals/ac17ef8e8dff4178bec0c736ae9a42b2
I20260924 16:02:30.673259 1763 log.cc:1196] T ac17ef8e8dff4178bec0c736ae9a42b2 P 23a7821f743042fb9fa18864c1487c4c: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265708917108-26036-0/minicluster-data/ts-2/wal/wals/ac17ef8e8dff4178bec0c736ae9a42b2
I20260924 16:02:30.673521 1763 ts_tablet_manager.cc:1939] T ac17ef8e8dff4178bec0c736ae9a42b2 P 23a7821f743042fb9fa18864c1487c4c: Deleting consensus metadata
I20260924 16:02:30.673534 1764 ts_tablet_manager.cc:1939] T ac17ef8e8dff4178bec0c736ae9a42b2 P 014fd818c4b34521b5e2587cf075b95a: Deleting consensus metadata
I20260924 16:02:30.674270 1154 catalog_manager.cc:4841] TS 23a7821f743042fb9fa18864c1487c4c (127.25.109.3:35417): tablet ac17ef8e8dff4178bec0c736ae9a42b2 (table test-table [id=e7fed9996f6e4d5695a5b17e1180181b]) successfully deleted
I20260924 16:02:30.674361 1155 catalog_manager.cc:4841] TS 014fd818c4b34521b5e2587cf075b95a (127.25.109.2:35349): tablet ac17ef8e8dff4178bec0c736ae9a42b2 (table test-table [id=e7fed9996f6e4d5695a5b17e1180181b]) successfully deleted
W20260924 16:02:30.677211 1154 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.109.1:36345 (error 108)
I20260924 16:02:30.677989 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 1386
I20260924 16:02:30.678344 1803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:30.677903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:36345 (local address 127.0.0.1:58778)
0924 16:02:30.678142 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:30.678220 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.1:36345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":74,"threads_started":1}
W20260924 16:02:30.678767 1154 catalog_manager.cc:4584] TS fd528086b31748cfa35d0ec7e5012efd (127.25.109.1:36345): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ac17ef8e8dff4178bec0c736ae9a42b2: Network error: Client connection negotiation failed: client connection to 127.25.109.1:36345: connect: Connection refused (error 111)
I20260924 16:02:30.685823 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 1526
I20260924 16:02:30.693044 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 1138
2026-09-24T16:02:30Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3823 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:02:30 dist-test-slave-n106 krb5kdc[1806](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:02:30 dist-test-slave-n106 krb5kdc[1806](info): set up 2 sockets
Sep 24 16:02:30 dist-test-slave-n106 krb5kdc[1806](info): commencing operation
krb5kdc: starting...
W20260924 16:02:31.734150 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:02:31 dist-test-slave-n106 krb5kdc[1806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:02:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:02:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
I20260924 16:02:31.855800 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:34103
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:39069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:34103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:31.968017 1819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:31.968329 1819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:31.968452 1819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:31.972970 1819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:02:31.973073 1819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:31.973111 1819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:31.973151 1819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:02:31.973186 1819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:02:31.978555 1819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:39069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:34103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:34103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:31.979713 1819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:31.980851 1819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:31.989588 1824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:31.989650 1828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:31.989748 1819 server_base.cc:1034] running on GCE node
W20260924 16:02:31.989646 1825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:31.990514 1819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:31.991678 1819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:31.992866 1819 hybrid_clock.cc:648] HybridClock initialized: now 1790265751992821 us; error 44 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T16:02:32Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1309 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:02:32 dist-test-slave-n106 krb5kdc[1833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:02:32 dist-test-slave-n106 krb5kdc[1833](info): set up 2 sockets
Sep 24 16:02:32 dist-test-slave-n106 krb5kdc[1833](info): commencing operation
krb5kdc: starting...
W20260924 16:02:33.050112 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:02:33 dist-test-slave-n106 krb5kdc[1833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:02:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:02:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
I20260924 16:02:33.175318 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:39381
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:39635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:39381
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:33.290005 1846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:33.290400 1846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:33.290490 1846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:33.294672 1846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:02:33.294777 1846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:33.294816 1846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:33.294852 1846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:02:33.294891 1846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:02:33.300478 1846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:39635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:39381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:39381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:33.301753 1846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:33.302879 1846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:33.311962 1852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:33.311954 1851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:33.312080 1846 server_base.cc:1034] running on GCE node
W20260924 16:02:33.311962 1854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:33.312893 1846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:33.314106 1846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:33.315275 1846 hybrid_clock.cc:648] HybridClock initialized: now 1790265753315259 us; error 28 us; skew 500 ppm
Sep 24 16:02:33 dist-test-slave-n106 krb5kdc[1833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265753, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:33.318282 1846 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:02:33.319258 1846 webserver.cc:460] Webserver started at http://127.25.109.62:44331/ using document root <none> and password file <none>
I20260924 16:02:33.319877 1846 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:33.319973 1846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:33.320226 1846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:33.322417 1846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "42088142d46f4782a1a8c6f8e1efc84f"
format_stamp: "Formatted at 2026-09-24 16:02:33 on dist-test-slave-n106"
server_key: "7ebca3c23aca170fdeead9b0f7d93c8c"
server_key_iv: "b4e2824323b31dc857d1562a3e39b57c"
server_key_version: "encryptionkey@0"
I20260924 16:02:33.322945 1846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "42088142d46f4782a1a8c6f8e1efc84f"
format_stamp: "Formatted at 2026-09-24 16:02:33 on dist-test-slave-n106"
server_key: "7ebca3c23aca170fdeead9b0f7d93c8c"
server_key_iv: "b4e2824323b31dc857d1562a3e39b57c"
server_key_version: "encryptionkey@0"
I20260924 16:02:33.326982 1846 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:02:33.329712 1861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:33.331123 1846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 16:02:33.331265 1846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "42088142d46f4782a1a8c6f8e1efc84f"
format_stamp: "Formatted at 2026-09-24 16:02:33 on dist-test-slave-n106"
server_key: "7ebca3c23aca170fdeead9b0f7d93c8c"
server_key_iv: "b4e2824323b31dc857d1562a3e39b57c"
server_key_version: "encryptionkey@0"
I20260924 16:02:33.331387 1846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:33.357527 1846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:33.361114 1846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:33.361327 1846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:33.368345 1846 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:39381
I20260924 16:02:33.368453 1913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:39381 every 8 connection(s)
I20260924 16:02:33.369396 1846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:02:33.372123 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 1846
I20260924 16:02:33.372270 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:02:33.372351 1914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:33.372663 26036 external_mini_cluster.cc:1428] Setting key 549689e810e03d25f4c0f39addf316a6
I20260924 16:02:33.379492 1914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f: Bootstrap starting.
Sep 24 16:02:33 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:02:33.382320 1914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:33.383055 1914 log.cc:824] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:33.384969 1914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f: No bootstrap required, opened a new log
I20260924 16:02:33.387537 1917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:33.374030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43184 (local address 127.25.109.62:39381)
0924 16:02:33.374459 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:33.374483 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:33.374522 (+ 39us) server_negotiation.cc:407] Connection header received
0924 16:02:33.375209 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:33.375228 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:33.375566 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:33.375850 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:33.376840 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:33.377862 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:33.379073 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:33.379538 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:33.379738 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:33.382460 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:33.382465 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:33.382479 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:33.382508 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:33.384204 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:33.384729 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:33.384735 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:33.384745 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:33.384855 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:33.385188 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:33.385191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:33.385193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:33.385705 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:33.385903 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:33.386164 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 16:02:33.386476 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":136,"threads_started":1}
I20260924 16:02:33.389212 1914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42088142d46f4782a1a8c6f8e1efc84f" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 39381 } }
I20260924 16:02:33.389430 1914 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:33.389487 1914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42088142d46f4782a1a8c6f8e1efc84f, State: Initialized, Role: FOLLOWER
I20260924 16:02:33.389922 1914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42088142d46f4782a1a8c6f8e1efc84f" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 39381 } }
I20260924 16:02:33.390062 1914 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:02:33.390127 1914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:02:33.390230 1914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:33.391100 1914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42088142d46f4782a1a8c6f8e1efc84f" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 39381 } }
I20260924 16:02:33.391458 1914 leader_election.cc:302] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42088142d46f4782a1a8c6f8e1efc84f; no voters:
I20260924 16:02:33.391755 1914 leader_election.cc:288] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:02:33.391917 1919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:33.392144 1919 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [term 1 LEADER]: Becoming Leader. State: Replica: 42088142d46f4782a1a8c6f8e1efc84f, State: Running, Role: LEADER
I20260924 16:02:33.392488 1919 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42088142d46f4782a1a8c6f8e1efc84f" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 39381 } }
I20260924 16:02:33.393031 1914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:02:33.394309 1920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42088142d46f4782a1a8c6f8e1efc84f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42088142d46f4782a1a8c6f8e1efc84f" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 39381 } } }
I20260924 16:02:33.394449 1920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:33.395606 1921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42088142d46f4782a1a8c6f8e1efc84f. Latest consensus state: current_term: 1 leader_uuid: "42088142d46f4782a1a8c6f8e1efc84f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42088142d46f4782a1a8c6f8e1efc84f" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 39381 } } }
I20260924 16:02:33.395848 1921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [sys.catalog]: This master's current role is: LEADER
W20260924 16:02:33.398078 1934 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:02:33.398146 1934 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:02:33.398222 1929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:02:33.398942 1929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:02:33.404623 1929 catalog_manager.cc:1347] Generated new cluster ID: 2fc077fea6724009a2a23be488c2ecb5
I20260924 16:02:33.404692 1929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:02:33.415782 1929 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:02:33.416721 1929 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:02:33.424710 1929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f: Generated new TSK 0
I20260924 16:02:33.425534 1929 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
I20260924 16:02:33.459837 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:0
--local_ip_for_outbound_sockets=127.25.109.1
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:39381
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:39635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:33.573547 1940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:33.573822 1940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:33.573865 1940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:33.577859 1940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:33.577914 1940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:33.577971 1940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:02:33.583130 1940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:39635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:39381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:33.584206 1940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:33.585278 1940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:33.594858 1945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:33.594864 1946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:33.594918 1940 server_base.cc:1034] running on GCE node
W20260924 16:02:33.594873 1948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:33.595543 1940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:33.596256 1940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:33.597419 1940 hybrid_clock.cc:648] HybridClock initialized: now 1790265753597409 us; error 27 us; skew 500 ppm
Sep 24 16:02:33 dist-test-slave-n106 krb5kdc[1833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265753, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:33.600100 1940 init.cc:377] Logged in from keytab as kudu/127.25.109.1@KRBTEST.COM (short username kudu)
I20260924 16:02:33.600970 1940 webserver.cc:460] Webserver started at http://127.25.109.1:43385/ using document root <none> and password file <none>
I20260924 16:02:33.601501 1940 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:33.601562 1940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:33.601730 1940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:33.603765 1940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-0/data/instance:
uuid: "4385e99def6d4f2fb8a0f2fb5d2d00e8"
format_stamp: "Formatted at 2026-09-24 16:02:33 on dist-test-slave-n106"
server_key: "d426c90e5a6bfc6fd417f4ec9403088d"
server_key_iv: "b94206687f7e041e8713fea3552afbb2"
server_key_version: "encryptionkey@0"
I20260924 16:02:33.604297 1940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance:
uuid: "4385e99def6d4f2fb8a0f2fb5d2d00e8"
format_stamp: "Formatted at 2026-09-24 16:02:33 on dist-test-slave-n106"
server_key: "d426c90e5a6bfc6fd417f4ec9403088d"
server_key_iv: "b94206687f7e041e8713fea3552afbb2"
server_key_version: "encryptionkey@0"
I20260924 16:02:33.608076 1940 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:02:33.610509 1955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:33.611711 1940 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:02:33.611842 1940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "4385e99def6d4f2fb8a0f2fb5d2d00e8"
format_stamp: "Formatted at 2026-09-24 16:02:33 on dist-test-slave-n106"
server_key: "d426c90e5a6bfc6fd417f4ec9403088d"
server_key_iv: "b94206687f7e041e8713fea3552afbb2"
server_key_version: "encryptionkey@0"
I20260924 16:02:33.611953 1940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:33.628473 1940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:33.631217 1940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:33.631409 1940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:33.631999 1940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:33.632884 1940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:33.632954 1940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:33.633031 1940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:33.633076 1940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:33.641928 1940 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:37897
I20260924 16:02:33.641947 2069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:37897 every 8 connection(s)
I20260924 16:02:33.642809 1940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:02:33.646481 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 1940
I20260924 16:02:33.646647 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance
I20260924 16:02:33.646947 26036 external_mini_cluster.cc:1428] Setting key fe0ce3247041d645fe3ddec6be2922a7
Sep 24 16:02:33 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265753, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.109.2@KRBTEST.COM; defaulting to no policy
I20260924 16:02:33.657745 1917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:33.644611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:59793 (local address 127.25.109.62:39381)
0924 16:02:33.644757 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:33.644760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:33.645572 (+ 812us) server_negotiation.cc:407] Connection header received
0924 16:02:33.647016 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:33.647020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:33.647083 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:33.647190 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:33.648365 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:33.648963 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:33.650473 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:33.650718 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:33.650804 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:33.653751 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:33.653757 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:33.653763 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:33.653801 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:33.655522 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:33.656057 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:33.656061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:33.656063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:33.656158 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:33.656537 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:33.656541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:33.656543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:33.656815 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:33.656927 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:33.657495 (+ 568us) server_negotiation.cc:299] Negotiation successful
0924 16:02:33.657614 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 16:02:33.658612 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:33.644987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.1:59793)
0924 16:02:33.645433 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:33.645475 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:33.646807 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:33.647372 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:33.647380 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:33.647736 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:33.648224 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:33.648237 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:33.649114 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:33.649118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:33.650345 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:33.650360 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:33.650884 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:33.650889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:33.651124 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:33.651793 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:33.651810 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:33.653543 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:33.655655 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:33.655660 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:33.655676 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:33.655946 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:33.656260 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:33.656262 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:33.656264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:33.656448 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:33.656921 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:33.656926 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:33.657223 (+ 297us) client_negotiation.cc:769] Sending connection context
0924 16:02:33.657391 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 16:02:33.657674 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.2@KRBTEST.COM" created.
I20260924 16:02:33.660037 2070 heartbeater.cc:347] Connected to a master server at 127.25.109.62:39381
I20260924 16:02:33.660315 2070 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:33.660883 2070 heartbeater.cc:511] Master 127.25.109.62:39381 requested a full tablet report, sending...
I20260924 16:02:33.662552 1878 ts_manager.cc:194] Registered new tserver with Master: 4385e99def6d4f2fb8a0f2fb5d2d00e8 (127.25.109.1:37897)
I20260924 16:02:33.664316 1878 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.1@KRBTEST.COM'} at 127.25.109.1:59793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
I20260924 16:02:33.677575 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:0
--local_ip_for_outbound_sockets=127.25.109.2
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:39381
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:39635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:33.791914 2075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:33.792217 2075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:33.792294 2075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:33.796607 2075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:33.796690 2075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:33.796787 2075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:02:33.802500 2075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:39635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:39381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:33.803685 2075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:33.804792 2075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:33.814760 2081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:33.814760 2083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:33.814760 2080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:33.815027 2075 server_base.cc:1034] running on GCE node
I20260924 16:02:33.815501 2075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:33.816222 2075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:33.817374 2075 hybrid_clock.cc:648] HybridClock initialized: now 1790265753817346 us; error 46 us; skew 500 ppm
Sep 24 16:02:33 dist-test-slave-n106 krb5kdc[1833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265753, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:33.820184 2075 init.cc:377] Logged in from keytab as kudu/127.25.109.2@KRBTEST.COM (short username kudu)
I20260924 16:02:33.821213 2075 webserver.cc:460] Webserver started at http://127.25.109.2:45501/ using document root <none> and password file <none>
I20260924 16:02:33.821800 2075 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:33.821887 2075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:33.822114 2075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:33.824236 2075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-1/data/instance:
uuid: "8fe3c429b1aa4e3eba767891505fcba0"
format_stamp: "Formatted at 2026-09-24 16:02:33 on dist-test-slave-n106"
server_key: "a3df43f95e52860bfbe4ce8b1777e419"
server_key_iv: "e4cdeef3a5a44ae6b97d1932ebb98603"
server_key_version: "encryptionkey@0"
I20260924 16:02:33.824877 2075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance:
uuid: "8fe3c429b1aa4e3eba767891505fcba0"
format_stamp: "Formatted at 2026-09-24 16:02:33 on dist-test-slave-n106"
server_key: "a3df43f95e52860bfbe4ce8b1777e419"
server_key_iv: "e4cdeef3a5a44ae6b97d1932ebb98603"
server_key_version: "encryptionkey@0"
I20260924 16:02:33.828668 2075 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:02:33.831243 2090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:33.832545 2075 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:02:33.832664 2075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "8fe3c429b1aa4e3eba767891505fcba0"
format_stamp: "Formatted at 2026-09-24 16:02:33 on dist-test-slave-n106"
server_key: "a3df43f95e52860bfbe4ce8b1777e419"
server_key_iv: "e4cdeef3a5a44ae6b97d1932ebb98603"
server_key_version: "encryptionkey@0"
I20260924 16:02:33.832772 2075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:33.852943 2075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:33.856338 2075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:33.856575 2075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:33.857218 2075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:33.858083 2075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:33.858152 2075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:33.858239 2075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:33.858287 2075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:33.867836 2075 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:36157
I20260924 16:02:33.867859 2204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:36157 every 8 connection(s)
I20260924 16:02:33.868906 2075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:02:33.874497 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 2075
I20260924 16:02:33.874660 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance
I20260924 16:02:33.875006 26036 external_mini_cluster.cc:1428] Setting key 89f569d37478ac21d1cee4a13d5dce33
Sep 24 16:02:33 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265753, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:02:33.885183 1917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:33.870904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:46795 (local address 127.25.109.62:39381)
0924 16:02:33.871053 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:33.871057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:33.871926 (+ 869us) server_negotiation.cc:407] Connection header received
0924 16:02:33.873389 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:33.873393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:33.873457 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:33.873547 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:33.875120 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:33.875759 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:33.877269 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:33.877690 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:33.877816 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:33.880827 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:33.880834 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:33.880841 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:33.880877 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:33.882698 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:33.883313 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:33.883318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:33.883320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:33.883444 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:33.883797 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:33.883801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:33.883803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:33.884070 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:33.884195 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:33.884809 (+ 614us) server_negotiation.cc:299] Negotiation successful
0924 16:02:33.884960 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:02:33.886040 2207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:33.871327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.2:46795)
0924 16:02:33.871765 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:33.871810 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:33.873158 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:33.873752 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:33.873759 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:33.874143 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:33.874933 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:33.874956 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:33.875907 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:33.875911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:33.877140 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:33.877150 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:33.877857 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:33.877860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:33.878038 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:33.878755 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:33.878782 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:33.880633 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:33.882848 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:33.882855 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:33.882871 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:33.883188 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:33.883546 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:33.883548 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:33.883550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:33.883692 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:33.884188 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:33.884194 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:33.884536 (+ 342us) client_negotiation.cc:769] Sending connection context
0924 16:02:33.884704 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 16:02:33.885005 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for kudu/127.25.109.3@KRBTEST.COM; defaulting to no policy
I20260924 16:02:33.887373 2205 heartbeater.cc:347] Connected to a master server at 127.25.109.62:39381
I20260924 16:02:33.887676 2205 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:33.888238 2205 heartbeater.cc:511] Master 127.25.109.62:39381 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.3@KRBTEST.COM" created.
I20260924 16:02:33.889492 1878 ts_manager.cc:194] Registered new tserver with Master: 8fe3c429b1aa4e3eba767891505fcba0 (127.25.109.2:36157)
I20260924 16:02:33.890364 1878 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.2@KRBTEST.COM'} at 127.25.109.2:46795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
I20260924 16:02:33.906957 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:0
--local_ip_for_outbound_sockets=127.25.109.3
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:39381
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:39635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:34.019351 2210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:34.019632 2210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:34.019676 2210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:34.023881 2210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:34.023947 2210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:34.024006 2210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:02:34.029353 2210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:39635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:39381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:34.030397 2210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:34.031401 2210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:34.041258 2216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:34.041258 2215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:34.041378 2210 server_base.cc:1034] running on GCE node
W20260924 16:02:34.041328 2218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:34.041965 2210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:34.042526 2210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:34.043707 2210 hybrid_clock.cc:648] HybridClock initialized: now 1790265754043687 us; error 29 us; skew 500 ppm
Sep 24 16:02:34 dist-test-slave-n106 krb5kdc[1833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265754, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:34.046356 2210 init.cc:377] Logged in from keytab as kudu/127.25.109.3@KRBTEST.COM (short username kudu)
I20260924 16:02:34.047155 2210 webserver.cc:460] Webserver started at http://127.25.109.3:44071/ using document root <none> and password file <none>
I20260924 16:02:34.047681 2210 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:34.047737 2210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:34.047899 2210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:34.049938 2210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-2/data/instance:
uuid: "9892089d7e5f42c7bb9857ec76217f45"
format_stamp: "Formatted at 2026-09-24 16:02:34 on dist-test-slave-n106"
server_key: "024e53c807ae8aa4ed047ee3aca1eae5"
server_key_iv: "c70f12c648eea6bfc6eae6b78e68eac2"
server_key_version: "encryptionkey@0"
I20260924 16:02:34.050460 2210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance:
uuid: "9892089d7e5f42c7bb9857ec76217f45"
format_stamp: "Formatted at 2026-09-24 16:02:34 on dist-test-slave-n106"
server_key: "024e53c807ae8aa4ed047ee3aca1eae5"
server_key_iv: "c70f12c648eea6bfc6eae6b78e68eac2"
server_key_version: "encryptionkey@0"
I20260924 16:02:34.054286 2210 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:02:34.056780 2225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:34.058023 2210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:02:34.058148 2210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "9892089d7e5f42c7bb9857ec76217f45"
format_stamp: "Formatted at 2026-09-24 16:02:34 on dist-test-slave-n106"
server_key: "024e53c807ae8aa4ed047ee3aca1eae5"
server_key_iv: "c70f12c648eea6bfc6eae6b78e68eac2"
server_key_version: "encryptionkey@0"
I20260924 16:02:34.058251 2210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:34.072726 2210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:34.075680 2210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:34.075878 2210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:34.076490 2210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:34.077333 2210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:34.077400 2210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:34.077476 2210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:34.077517 2210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:34.086449 2210 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:42523
I20260924 16:02:34.086475 2339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:42523 every 8 connection(s)
I20260924 16:02:34.087416 2210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:02:34.093323 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 2210
I20260924 16:02:34.093479 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance
I20260924 16:02:34.093845 26036 external_mini_cluster.cc:1428] Setting key 286479e22d84a08ec72e54c9868bc0cf
Sep 24 16:02:34 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265754, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:02:34.101370 1917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.089276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:44409 (local address 127.25.109.62:39381)
0924 16:02:34.089409 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:34.089414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:34.090241 (+ 827us) server_negotiation.cc:407] Connection header received
0924 16:02:34.091559 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:34.091562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:34.091611 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:34.091695 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:34.092843 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.093572 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.095266 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.095610 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.095717 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:34.098132 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:34.098135 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:34.098137 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:34.098163 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:34.099440 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:34.099943 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:34.099946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:34.099947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:34.100027 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:34.100314 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:34.100316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:34.100317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:34.100533 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:34.100615 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:34.101126 (+ 511us) server_negotiation.cc:299] Negotiation successful
0924 16:02:34.101236 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 16:02:34.102178 2342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.089669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.3:44409)
0924 16:02:34.090093 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.090137 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:34.091341 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:34.091859 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:34.091865 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:34.092204 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:34.092698 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:34.092711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.093725 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.093729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:34.095126 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:34.095139 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.095745 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.095747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:34.095952 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:34.096601 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:34.096625 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:34.097976 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:34.099580 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:34.099585 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:34.099596 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:34.099855 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:34.100103 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:34.100105 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:34.100107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:34.100233 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:34.100638 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:34.100644 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:34.100876 (+ 232us) client_negotiation.cc:769] Sending connection context
0924 16:02:34.101040 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 16:02:34.101294 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":102,"threads_started":1}
I20260924 16:02:34.103458 2340 heartbeater.cc:347] Connected to a master server at 127.25.109.62:39381
I20260924 16:02:34.103705 2340 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:34.104202 2340 heartbeater.cc:511] Master 127.25.109.62:39381 requested a full tablet report, sending...
I20260924 16:02:34.105345 1878 ts_manager.cc:194] Registered new tserver with Master: 9892089d7e5f42c7bb9857ec76217f45 (127.25.109.3:42523)
I20260924 16:02:34.105913 1878 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.3@KRBTEST.COM'} at 127.25.109.3:44409
I20260924 16:02:34.107759 26036 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.25.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.1.keytab.
Entry for principal oryx/127.25.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.1.keytab.
WARNING: no policy specified for oryx/127.25.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.2.keytab.
Entry for principal oryx/127.25.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.2.keytab.
WARNING: no policy specified for oryx/127.25.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.3.keytab.
Entry for principal oryx/127.25.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.3.keytab.
I20260924 16:02:34.183073 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 1940
I20260924 16:02:34.189491 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 2075
I20260924 16:02:34.195346 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 2210
I20260924 16:02:34.201174 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 1846
I20260924 16:02:34.207604 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:39381
--webserver_interface=127.25.109.62
--webserver_port=44331
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:39635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:39381
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:34.319234 2349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:34.319563 2349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:34.319644 2349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:34.323804 2349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:02:34.323889 2349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:34.323926 2349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:34.323979 2349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:02:34.324018 2349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:02:34.329286 2349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:39635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:39381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:39381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=44331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:34.330387 2349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:34.331429 2349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:34.340082 2357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:34.340073 2354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:34.340075 2355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:34.340206 2349 server_base.cc:1034] running on GCE node
I20260924 16:02:34.340996 2349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:34.342067 2349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:34.343266 2349 hybrid_clock.cc:648] HybridClock initialized: now 1790265754343252 us; error 35 us; skew 500 ppm
Sep 24 16:02:34 dist-test-slave-n106 krb5kdc[1833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265754, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:34.346091 2349 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:02:34.346961 2349 webserver.cc:460] Webserver started at http://127.25.109.62:44331/ using document root <none> and password file <none>
I20260924 16:02:34.347549 2349 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:34.347651 2349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:34.351605 2349 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:02:34.353979 2364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:34.355253 2349 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:02:34.355407 2349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "42088142d46f4782a1a8c6f8e1efc84f"
format_stamp: "Formatted at 2026-09-24 16:02:33 on dist-test-slave-n106"
server_key: "7ebca3c23aca170fdeead9b0f7d93c8c"
server_key_iv: "b4e2824323b31dc857d1562a3e39b57c"
server_key_version: "encryptionkey@0"
I20260924 16:02:34.355993 2349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:34.371155 2349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:34.374127 2349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:34.374336 2349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:34.381544 2349 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:39381
I20260924 16:02:34.381564 2417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:39381 every 8 connection(s)
I20260924 16:02:34.382651 2349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:02:34.383563 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 2349
I20260924 16:02:34.383996 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:37897
--local_ip_for_outbound_sockets=127.25.109.1
--tserver_master_addrs=127.25.109.62:39381
--webserver_port=43385
--webserver_interface=127.25.109.1
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:39635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.25.109.1
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:02:34.387867 2418 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:02:34.390661 2418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f: Bootstrap starting.
I20260924 16:02:34.411461 2418 log.cc:824] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:34.418231 2418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:02:34.418610 2418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f: Bootstrap complete.
I20260924 16:02:34.424288 2418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42088142d46f4782a1a8c6f8e1efc84f" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 39381 } }
I20260924 16:02:34.424813 2418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42088142d46f4782a1a8c6f8e1efc84f, State: Initialized, Role: FOLLOWER
I20260924 16:02:34.425336 2418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42088142d46f4782a1a8c6f8e1efc84f" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 39381 } }
I20260924 16:02:34.425470 2418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:02:34.425523 2418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:02:34.425603 2418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [term 1 FOLLOWER]: Advancing to term 2
I20260924 16:02:34.428043 2418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42088142d46f4782a1a8c6f8e1efc84f" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 39381 } }
I20260924 16:02:34.428393 2418 leader_election.cc:302] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42088142d46f4782a1a8c6f8e1efc84f; no voters:
I20260924 16:02:34.428784 2418 leader_election.cc:288] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 16:02:34.428879 2422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [term 2 FOLLOWER]: Leader election won for term 2
I20260924 16:02:34.429073 2422 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [term 2 LEADER]: Becoming Leader. State: Replica: 42088142d46f4782a1a8c6f8e1efc84f, State: Running, Role: LEADER
I20260924 16:02:34.429378 2422 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42088142d46f4782a1a8c6f8e1efc84f" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 39381 } }
I20260924 16:02:34.430059 2418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:02:34.430343 2422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "42088142d46f4782a1a8c6f8e1efc84f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42088142d46f4782a1a8c6f8e1efc84f" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 39381 } } }
I20260924 16:02:34.430454 2422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:34.430637 2422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42088142d46f4782a1a8c6f8e1efc84f. Latest consensus state: current_term: 2 leader_uuid: "42088142d46f4782a1a8c6f8e1efc84f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42088142d46f4782a1a8c6f8e1efc84f" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 39381 } } }
I20260924 16:02:34.430707 2422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:34.435250 2437 catalog_manager.cc:1259] Loaded cluster ID: 2fc077fea6724009a2a23be488c2ecb5
I20260924 16:02:34.435309 2437 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f: loading cluster ID for follower catalog manager: success
I20260924 16:02:34.437737 2437 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f: acquiring CA information for follower catalog manager: success
I20260924 16:02:34.439216 2437 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:02:34.439719 2426 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:02:34.440794 2426 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:02:34.441274 2426 catalog_manager.cc:1259] Loaded cluster ID: 2fc077fea6724009a2a23be488c2ecb5
I20260924 16:02:34.441331 2426 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:02:34.442006 2426 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:02:34.442492 2426 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 42088142d46f4782a1a8c6f8e1efc84f: Loaded TSK: 0
I20260924 16:02:34.443157 2426 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 16:02:34.529245 2420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:34.529546 2420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:34.529618 2420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:34.534237 2420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:34.534317 2420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:34.534400 2420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:02:34.539970 2420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:39635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.1.keytab
--principal=oryx/127.25.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:37897
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=43385
--tserver_master_addrs=127.25.109.62:39381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:34.541076 2420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:34.542085 2420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:34.551699 2447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:34.551699 2444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:34.551685 2445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:34.552021 2420 server_base.cc:1034] running on GCE node
I20260924 16:02:34.552548 2420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:34.553184 2420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:34.554358 2420 hybrid_clock.cc:648] HybridClock initialized: now 1790265754554343 us; error 30 us; skew 500 ppm
Sep 24 16:02:34 dist-test-slave-n106 krb5kdc[1833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265754, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:34.557046 2420 init.cc:377] Logged in from keytab as oryx/127.25.109.1@KRBTEST.COM (short username oryx)
I20260924 16:02:34.557996 2420 webserver.cc:460] Webserver started at http://127.25.109.1:43385/ using document root <none> and password file <none>
I20260924 16:02:34.558562 2420 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:34.558635 2420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:34.562772 2420 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:02:34.564958 2454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:34.566239 2420 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:02:34.566335 2420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "4385e99def6d4f2fb8a0f2fb5d2d00e8"
format_stamp: "Formatted at 2026-09-24 16:02:33 on dist-test-slave-n106"
server_key: "d426c90e5a6bfc6fd417f4ec9403088d"
server_key_iv: "b94206687f7e041e8713fea3552afbb2"
server_key_version: "encryptionkey@0"
I20260924 16:02:34.566919 2420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:34.595542 2420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:34.599078 2420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:34.599397 2420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:34.600111 2420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:34.601046 2420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:34.601095 2420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:34.601166 2420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:34.601197 2420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:34.610554 2420 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:37897
I20260924 16:02:34.610641 2570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:37897 every 8 connection(s)
I20260924 16:02:34.611702 2420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:02:34.615345 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 2420
I20260924 16:02:34.615797 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:36157
--local_ip_for_outbound_sockets=127.25.109.2
--tserver_master_addrs=127.25.109.62:39381
--webserver_port=45501
--webserver_interface=127.25.109.2
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:39635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.25.109.2
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:02:34 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:34 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 16:02:34.626574 2573 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:34.613948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:37847 (local address 127.25.109.62:39381)
0924 16:02:34.614457 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:34.614477 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:34.614709 (+ 232us) server_negotiation.cc:407] Connection header received
0924 16:02:34.616490 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:34.616503 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:34.616830 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:34.617176 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:34.618436 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.619443 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.620668 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.621208 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.621390 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:34.625761 (+ 4371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.1:37847: BlockingRecv error: recv got EOF from 127.25.109.1:37847 (error 108)
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":108,"threads_started":1}
I20260924 16:02:34.626585 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.614035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.1:37847)
0924 16:02:34.614548 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.614592 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:34.615967 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:34.617262 (+ 1295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:34.617270 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:34.617750 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:34.618243 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:34.618258 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.619596 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.619600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:34.620541 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:34.620552 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.621480 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.621490 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:34.621879 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:34.622708 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:34.622738 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:34.625510 (+ 2772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":164,"threads_started":1}
W20260924 16:02:34.627447 2571 heartbeater.cc:644] Failed to heartbeat to 127.25.109.62:39381 (0 consecutive failures): Not authorized: Failed to ping master at 127.25.109.62:39381: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Sep 24 16:02:34 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:34 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:34.634322 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.627927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.1:58425)
0924 16:02:34.628176 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.628188 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:34.628268 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:34.628570 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:34.628572 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:34.628712 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:34.628811 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:34.628817 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.629903 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.629907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:34.630550 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:34.630560 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.631170 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.631176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:34.631264 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:34.631904 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:34.631917 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:34.634112 (+ 2195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179}
W20260924 16:02:34.634367 2573 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:34.628049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:58425 (local address 127.25.109.62:39381)
0924 16:02:34.628183 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:34.628189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:34.628229 (+ 40us) server_negotiation.cc:407] Connection header received
0924 16:02:34.628356 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:34.628359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:34.628450 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:34.628555 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:34.628919 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.629788 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.630656 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.631019 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.631118 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:34.634267 (+ 3149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.1:58425: BlockingRecv error: recv got EOF from 127.25.109.1:58425 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 16:02:34 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:34 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:34.640610 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.635249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.1:50631)
0924 16:02:34.635460 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.635472 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:34.635542 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:34.635781 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:34.635783 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:34.635911 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:34.635994 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:34.635999 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.637039 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.637042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:34.637553 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:34.637561 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.638123 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.638125 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:34.638185 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:34.638674 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:34.638685 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:34.640509 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135}
W20260924 16:02:34.640756 2573 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:34.635375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:50631 (local address 127.25.109.62:39381)
0924 16:02:34.635500 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:34.635504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:34.635525 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:02:34.635627 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:34.635630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:34.635676 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:34.635744 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:34.636091 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.636922 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.637655 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.638020 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.638120 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:34.640641 (+ 2521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.1:50631: BlockingRecv error: recv got EOF from 127.25.109.1:50631 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 16:02:34.640900 2571 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 16:02:34.736967 2575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:34.737219 2575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:34.737265 2575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:34.741222 2575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:34.741302 2575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:34.741385 2575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:02:34.746758 2575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:39635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.2.keytab
--principal=oryx/127.25.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:36157
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=45501
--tserver_master_addrs=127.25.109.62:39381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:34.747884 2575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:34.749002 2575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:34.758955 2580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:34.758996 2583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:34.758956 2581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:34.759191 2575 server_base.cc:1034] running on GCE node
I20260924 16:02:34.759701 2575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:34.760473 2575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:34.761662 2575 hybrid_clock.cc:648] HybridClock initialized: now 1790265754761645 us; error 35 us; skew 500 ppm
Sep 24 16:02:34 dist-test-slave-n106 krb5kdc[1833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265754, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:34.764349 2575 init.cc:377] Logged in from keytab as oryx/127.25.109.2@KRBTEST.COM (short username oryx)
I20260924 16:02:34.765260 2575 webserver.cc:460] Webserver started at http://127.25.109.2:45501/ using document root <none> and password file <none>
I20260924 16:02:34.765843 2575 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:34.765946 2575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:34.770042 2575 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260924 16:02:34.772365 2590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:34.773778 2575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:02:34.773943 2575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "8fe3c429b1aa4e3eba767891505fcba0"
format_stamp: "Formatted at 2026-09-24 16:02:33 on dist-test-slave-n106"
server_key: "a3df43f95e52860bfbe4ce8b1777e419"
server_key_iv: "e4cdeef3a5a44ae6b97d1932ebb98603"
server_key_version: "encryptionkey@0"
I20260924 16:02:34.774540 2575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:34.794486 2575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:34.797761 2575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:34.797971 2575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:34.798593 2575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:34.799501 2575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:34.799571 2575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:34.799638 2575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:34.799690 2575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:34.808879 2575 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:36157
I20260924 16:02:34.808915 2703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:36157 every 8 connection(s)
I20260924 16:02:34.809921 2575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:02:34.812670 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 2575
I20260924 16:02:34.813212 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:42523
--local_ip_for_outbound_sockets=127.25.109.3
--tserver_master_addrs=127.25.109.62:39381
--webserver_port=44071
--webserver_interface=127.25.109.3
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:39635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.25.109.3
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:02:34 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:34 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 16:02:34.823947 2573 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:34.811900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:53131 (local address 127.25.109.62:39381)
0924 16:02:34.812055 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:34.812060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:34.813188 (+ 1128us) server_negotiation.cc:407] Connection header received
0924 16:02:34.815222 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:34.815228 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:34.815302 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:34.815409 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:34.816872 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.817424 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.819399 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.819630 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.819720 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:34.823813 (+ 4093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.2:53131: BlockingRecv error: recv got EOF from 127.25.109.2:53131 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 16:02:34.824687 2706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.812367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.2:53131)
0924 16:02:34.812998 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.813062 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:34.814989 (+ 1927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:34.815586 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:34.815592 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:34.816044 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:34.816705 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:34.816731 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.817563 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.817570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:34.819259 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:34.819277 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.819810 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.819816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:34.820089 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:34.820892 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:34.820920 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:34.823674 (+ 2754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":112,"threads_started":1}
W20260924 16:02:34.825321 2704 heartbeater.cc:644] Failed to heartbeat to 127.25.109.62:39381 (0 consecutive failures): Not authorized: Failed to ping master at 127.25.109.62:39381: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Sep 24 16:02:34 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:34 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:34.831655 2706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.825819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.2:57621)
0924 16:02:34.826015 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.826030 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:34.826116 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:34.826400 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:34.826403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:34.826579 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:34.826695 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:34.826701 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.827394 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.827397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:34.828334 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:34.828345 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.828930 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.828934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:34.829042 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:34.829514 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:34.829529 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:34.831512 (+ 1983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115}
W20260924 16:02:34.831831 2573 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:34.825911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:57621 (local address 127.25.109.62:39381)
0924 16:02:34.826023 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:34.826026 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:34.826070 (+ 44us) server_negotiation.cc:407] Connection header received
0924 16:02:34.826240 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:34.826243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:34.826298 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:34.826362 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:34.826810 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.827302 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.828594 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.828821 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.828903 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:34.831680 (+ 2777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.2:57621: BlockingRecv error: recv got EOF from 127.25.109.2:57621 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 16:02:34 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:34 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:34.837486 2708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:34.832186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.2:41343)
0924 16:02:34.832575 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:34.832590 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:34.832708 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:34.832927 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:34.832929 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:34.833115 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:34.833209 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:34.833214 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.833908 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.833913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:34.834502 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:34.834510 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.834961 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.834964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:34.835074 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:34.835589 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:34.835601 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:34.837299 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":123,"threads_started":1}
W20260924 16:02:34.837520 2573 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:34.832239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:41343 (local address 127.25.109.62:39381)
0924 16:02:34.832358 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:34.832364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:34.832667 (+ 303us) server_negotiation.cc:407] Connection header received
0924 16:02:34.832784 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:34.832787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:34.832828 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:34.832911 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:34.833315 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.833808 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.834626 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:34.834843 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:34.834949 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:34.837412 (+ 2463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.2:41343: BlockingRecv error: recv got EOF from 127.25.109.2:41343 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260924 16:02:34.837864 2704 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 16:02:34.932209 2707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:34.932538 2707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:34.932585 2707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:34.936684 2707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:34.936739 2707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:34.936816 2707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:02:34.942096 2707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:39635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/krb5kdc/oryx_127.25.109.3.keytab
--principal=oryx/127.25.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:42523
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=44071
--tserver_master_addrs=127.25.109.62:39381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:34.943166 2707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:34.944221 2707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:34.953753 2714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:34.953801 2716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:34.953864 2707 server_base.cc:1034] running on GCE node
W20260924 16:02:34.953748 2713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:34.954409 2707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:34.955224 2707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:34.956465 2707 hybrid_clock.cc:648] HybridClock initialized: now 1790265754956440 us; error 35 us; skew 500 ppm
Sep 24 16:02:34 dist-test-slave-n106 krb5kdc[1833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265754, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:34.959192 2707 init.cc:377] Logged in from keytab as oryx/127.25.109.3@KRBTEST.COM (short username oryx)
I20260924 16:02:34.960189 2707 webserver.cc:460] Webserver started at http://127.25.109.3:44071/ using document root <none> and password file <none>
I20260924 16:02:34.960868 2707 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:34.960971 2707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:34.964838 2707 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:02:34.966955 2723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:34.968183 2707 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:02:34.968313 2707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "9892089d7e5f42c7bb9857ec76217f45"
format_stamp: "Formatted at 2026-09-24 16:02:34 on dist-test-slave-n106"
server_key: "024e53c807ae8aa4ed047ee3aca1eae5"
server_key_iv: "c70f12c648eea6bfc6eae6b78e68eac2"
server_key_version: "encryptionkey@0"
I20260924 16:02:34.968922 2707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:34.983376 2707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:34.986349 2707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:34.986564 2707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:34.987172 2707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:34.988023 2707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:34.988092 2707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:34.988163 2707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:34.988204 2707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:34.997571 2707 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:42523
I20260924 16:02:34.997613 2838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:42523 every 8 connection(s)
I20260924 16:02:34.998606 2707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:02:35.000852 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 2707
Sep 24 16:02:35 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:35 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 16:02:35.011197 2573 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:35.000522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:42559 (local address 127.25.109.62:39381)
0924 16:02:35.000733 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:35.000738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:35.001780 (+ 1042us) server_negotiation.cc:407] Connection header received
0924 16:02:35.003697 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:35.003700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:35.003751 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:35.003828 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:35.005408 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.005943 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.007118 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.007326 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.007399 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:35.011043 (+ 3644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.3:42559: BlockingRecv error: recv got EOF from 127.25.109.3:42559 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
I20260924 16:02:35.011853 2843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:35.001915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42702 (local address 127.25.109.62:39381)
0924 16:02:35.002133 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:35.002143 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:35.002156 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:02:35.002199 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:35.002201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:35.002244 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:35.002324 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:35.003392 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.003986 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.005225 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.005473 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.005615 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:35.007115 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:35.007125 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:35.007143 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:35.007177 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:35.009209 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:35.009679 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:35.009685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:35.009691 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:35.009811 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:35.010106 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:35.010110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:35.010112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:35.010626 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:35.010838 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:35.011099 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 16:02:35.011712 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":86,"threads_started":1}
I20260924 16:02:35.011991 2841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:35.001091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.3:42559)
0924 16:02:35.001612 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:35.001675 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:35.003489 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:35.004006 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:35.004013 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:35.004570 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:35.005275 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:35.005292 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.006070 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.006073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:35.006996 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:35.007010 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.007496 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.007501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:35.007710 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:35.008182 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:35.008207 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:35.010947 (+ 2740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":113,"threads_started":1}
W20260924 16:02:35.012528 2839 heartbeater.cc:644] Failed to heartbeat to 127.25.109.62:39381 (0 consecutive failures): Not authorized: Failed to ping master at 127.25.109.62:39381: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Sep 24 16:02:35 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:35 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:35.017916 2841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:35.012953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.3:36953)
0924 16:02:35.013068 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:35.013082 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:35.013171 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:35.013419 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:35.013422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:35.013569 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:35.013696 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:35.013707 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.014560 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.014563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:35.015162 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:35.015171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.015640 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.015642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:35.015706 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:35.016108 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:35.016125 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:35.017820 (+ 1695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 16:02:35.018041 2843 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:35.013016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:36953 (local address 127.25.109.62:39381)
0924 16:02:35.013126 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:35.013130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:35.013142 (+ 12us) server_negotiation.cc:407] Connection header received
0924 16:02:35.013282 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:35.013285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:35.013328 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:35.013404 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:35.013815 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.014458 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.015284 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.015540 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.015629 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:35.017935 (+ 2306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.3:36953: BlockingRecv error: recv got EOF from 127.25.109.3:36953 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 16:02:35 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:35 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:35.022761 2841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:35.018443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.3:56809)
0924 16:02:35.018563 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:35.018578 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:35.018646 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:35.018820 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:35.018822 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:35.018938 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:35.019013 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:35.019018 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.019648 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.019650 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:35.020157 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:35.020164 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.020597 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.020600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:35.020656 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:35.021022 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:35.021032 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:35.022671 (+ 1639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 16:02:35.022902 2843 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:35.018517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:56809 (local address 127.25.109.62:39381)
0924 16:02:35.018641 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:35.018644 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:35.018655 (+ 11us) server_negotiation.cc:407] Connection header received
0924 16:02:35.018703 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:35.018705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:35.018736 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:35.018797 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:35.019097 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.019555 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.020248 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.020499 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.020571 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:35.022795 (+ 2224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.3:56809: BlockingRecv error: recv got EOF from 127.25.109.3:56809 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260924 16:02:35.023067 2839 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 16:02:35 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:35 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:35.647102 2844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:35.641587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.1:60193)
0924 16:02:35.641885 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:35.641899 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:35.641990 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:35.642284 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:35.642287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:35.642499 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:35.642603 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:35.642608 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.643447 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.643450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:35.644049 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:35.644057 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.644541 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.644543 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:35.644612 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:35.645190 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:35.645203 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:35.646975 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":91,"threads_started":1}
W20260924 16:02:35.647305 2845 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:35.641656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:60193 (local address 127.25.109.62:39381)
0924 16:02:35.641975 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:35.641979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:35.641994 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:02:35.642081 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:35.642083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:35.642128 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:35.642212 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:35.642729 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.643324 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.644173 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.644426 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.644518 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:35.647121 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.1:60193: BlockingRecv error: recv got EOF from 127.25.109.1:60193 (error 108)
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":105,"threads_started":1}
Sep 24 16:02:35 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:35 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:35.844413 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:35.838511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.2:45319)
0924 16:02:35.838837 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:35.838850 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:35.838987 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:35.839278 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:35.839281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:35.839509 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:35.839606 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:35.839613 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.840396 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.840401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:35.841051 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:35.841061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.841531 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.841535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:35.841642 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:35.842136 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:35.842147 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:35.844228 (+ 2081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":77,"threads_started":1}
W20260924 16:02:35.844519 2845 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:35.838581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:45319 (local address 127.25.109.62:39381)
0924 16:02:35.838754 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:35.838760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:35.838945 (+ 185us) server_negotiation.cc:407] Connection header received
0924 16:02:35.839099 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:35.839103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:35.839157 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:35.839251 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:35.839734 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.840277 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.841181 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:35.841410 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:35.841526 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:35.844371 (+ 2845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.2:45319: BlockingRecv error: recv got EOF from 127.25.109.2:45319 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 24 16:02:36 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:36 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:36.029685 2847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:36.023858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.3:44263)
0924 16:02:36.024230 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:36.024247 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:36.024425 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:36.024708 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:36.024711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:36.024937 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:36.025035 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:36.025041 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:36.025938 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:36.025943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:36.026512 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:36.026520 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:36.026966 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:36.026970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:36.027054 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:36.027598 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:36.027610 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:36.029489 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":126,"threads_started":1}
W20260924 16:02:36.029719 2845 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:36.024003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:44263 (local address 127.25.109.62:39381)
0924 16:02:36.024205 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:36.024210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:36.024337 (+ 127us) server_negotiation.cc:407] Connection header received
0924 16:02:36.024533 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:36.024536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:36.024589 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:36.024689 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:36.025160 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:36.025823 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:36.026631 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:36.026856 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:36.026962 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:36.029613 (+ 2651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.3:44263: BlockingRecv error: recv got EOF from 127.25.109.3:44263 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 24 16:02:36 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:36 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:36.653641 2848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:36.647926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.1:58479)
0924 16:02:36.648206 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:36.648224 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:36.648332 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:36.648605 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:36.648607 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:36.648792 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:36.648895 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:36.648900 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:36.649688 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:36.649691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:36.650297 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:36.650306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:36.650792 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:36.650795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:36.650880 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:36.651445 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:36.651457 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:36.653434 (+ 1977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":76,"threads_started":1}
W20260924 16:02:36.653698 2849 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:36.648063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:58479 (local address 127.25.109.62:39381)
0924 16:02:36.648312 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:36.648316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:36.648330 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:02:36.648439 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:36.648441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:36.648497 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:36.648566 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:36.649007 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:36.649547 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:36.650430 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:36.650668 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:36.650775 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:36.653568 (+ 2793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.1:58479: BlockingRecv error: recv got EOF from 127.25.109.1:58479 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":84,"threads_started":1}
Sep 24 16:02:36 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:36 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:36.850805 2850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:36.845330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.2:60085)
0924 16:02:36.845598 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:36.845611 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:36.845708 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:36.845943 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:36.845945 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:36.846124 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:36.846220 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:36.846225 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:36.847096 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:36.847099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:36.847820 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:36.847830 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:36.848240 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:36.848243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:36.848316 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:36.848884 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:36.848896 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:36.850669 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":71,"threads_started":1}
W20260924 16:02:36.851011 2849 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:36.845462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:60085 (local address 127.25.109.62:39381)
0924 16:02:36.845627 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:36.845631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:36.845659 (+ 28us) server_negotiation.cc:407] Connection header received
0924 16:02:36.845794 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:36.845796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:36.845840 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:36.845906 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:36.846332 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:36.846968 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:36.847921 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:36.848147 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:36.848226 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:36.850860 (+ 2634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.2:60085: BlockingRecv error: recv got EOF from 127.25.109.2:60085 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 24 16:02:37 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:37 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:37.048813 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:37.030575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.3:35893)
0924 16:02:37.030835 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:37.030849 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:37.030947 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:37.031200 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:37.031202 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:37.031376 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:37.031580 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:37.031592 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:37.032651 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:37.032657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:37.033286 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:37.033296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:37.033710 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:37.033713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:37.033774 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:37.046630 (+ 12856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:37.046643 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:37.048664 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":89,"threads_started":1}
W20260924 16:02:37.048964 2849 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:37.030725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:35893 (local address 127.25.109.62:39381)
0924 16:02:37.030853 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:37.030864 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:37.030897 (+ 33us) server_negotiation.cc:407] Connection header received
0924 16:02:37.031030 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:37.031032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:37.031077 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:37.031151 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:37.031691 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:37.032528 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:37.033395 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:37.033603 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:37.033687 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:37.048812 (+ 15125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.3:35893: BlockingRecv error: recv got EOF from 127.25.109.3:35893 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 16:02:37 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:37 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:37.660207 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:37.654506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.1:49131)
0924 16:02:37.654788 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:37.654802 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:37.654892 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:37.655147 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:37.655149 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:37.655323 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:37.655417 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:37.655422 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:37.656179 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:37.656183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:37.657046 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:37.657060 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:37.657608 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:37.657612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:37.657683 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:37.658162 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:37.658173 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:37.660017 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
W20260924 16:02:37.660296 2853 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:37.654603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:49131 (local address 127.25.109.62:39381)
0924 16:02:37.654843 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:37.654847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:37.654860 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:02:37.654976 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:37.654978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:37.655031 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:37.655105 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:37.655516 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:37.656046 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:37.657157 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:37.657505 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:37.657586 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:37.660140 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.1:49131: BlockingRecv error: recv got EOF from 127.25.109.1:49131 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":76,"threads_started":1}
Sep 24 16:02:37 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:37 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:37.857980 2854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:37.851610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.2:43875)
0924 16:02:37.852042 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:37.852056 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:37.852205 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:37.852545 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:37.852549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:37.852773 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:37.852873 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:37.852879 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:37.853735 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:37.853741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:37.854404 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:37.854414 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:37.854937 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:37.854941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:37.855036 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:37.855685 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:37.855698 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:37.857781 (+ 2083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":104,"threads_started":1}
W20260924 16:02:37.858044 2853 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:37.851689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:43875 (local address 127.25.109.62:39381)
0924 16:02:37.851852 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:37.851859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:37.852145 (+ 286us) server_negotiation.cc:407] Connection header received
0924 16:02:37.852316 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:37.852320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:37.852380 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:37.852516 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:37.853023 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:37.853612 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:37.854541 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:37.854792 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:37.854925 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:37.857924 (+ 2999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.2:43875: BlockingRecv error: recv got EOF from 127.25.109.2:43875 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 16:02:38 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:38 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:38.054919 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:38.049609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.3:35703)
0924 16:02:38.049897 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:38.049911 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:38.050024 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:38.050258 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:38.050260 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:38.050436 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:38.050531 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:38.050536 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:38.051250 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:38.051252 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:38.051797 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:38.051804 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:38.052252 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:38.052254 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:38.052316 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:38.052900 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:38.052911 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:38.054770 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":93,"threads_started":1}
W20260924 16:02:38.055058 2853 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:38.049669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:35703 (local address 127.25.109.62:39381)
0924 16:02:38.049815 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:38.049820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:38.049978 (+ 158us) server_negotiation.cc:407] Connection header received
0924 16:02:38.050102 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:38.050105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:38.050152 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:38.050219 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:38.050631 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:38.051159 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:38.051916 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:38.052150 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:38.052239 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:38.054927 (+ 2688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.3:35703: BlockingRecv error: recv got EOF from 127.25.109.3:35703 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 16:02:38 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:38 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:38.666637 2856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:38.661001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.1:36325)
0924 16:02:38.661280 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:38.661293 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:38.661381 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:38.661614 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:38.661616 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:38.661806 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:38.661976 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:38.661997 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:38.662797 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:38.662802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:38.663530 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:38.663540 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:38.664014 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:38.664018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:38.664104 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:38.664678 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:38.664689 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:38.666456 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":73,"threads_started":1}
W20260924 16:02:38.666700 2857 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:38.661055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:36325 (local address 127.25.109.62:39381)
0924 16:02:38.661365 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:38.661369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:38.661383 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:02:38.661475 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:38.661478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:38.661525 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:38.661606 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:38.662142 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:38.662683 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:38.663656 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:38.663900 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:38.664008 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:38.666581 (+ 2573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.1:36325: BlockingRecv error: recv got EOF from 127.25.109.1:36325 (error 108)
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":123,"threads_started":1}
Sep 24 16:02:38 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:38 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:38.864130 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:38.858766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.2:49071)
0924 16:02:38.859191 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:38.859208 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:38.859314 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:38.859544 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:38.859546 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:38.859757 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:38.859853 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:38.859859 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:38.860726 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:38.860729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:38.861330 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:38.861338 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:38.861731 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:38.861734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:38.861795 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:38.862257 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:38.862268 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:38.864014 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":191,"threads_started":1}
W20260924 16:02:38.864312 2857 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:38.858835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:49071 (local address 127.25.109.62:39381)
0924 16:02:38.859129 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:38.859133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:38.859285 (+ 152us) server_negotiation.cc:407] Connection header received
0924 16:02:38.859393 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:38.859396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:38.859443 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:38.859516 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:38.859951 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:38.860591 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:38.861425 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:38.861639 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:38.861713 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:38.864155 (+ 2442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.2:49071: BlockingRecv error: recv got EOF from 127.25.109.2:49071 (error 108)
Metrics: {"server-negotiator.queue_time_us":178}
Sep 24 16:02:39 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:39 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:39.061604 2859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:39.055778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.3:33035)
0924 16:02:39.056112 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:39.056125 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:39.056267 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:39.056558 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:39.056562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:39.056777 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:39.056869 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:39.056875 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:39.057635 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:39.057640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:39.058202 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:39.058210 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:39.058728 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:39.058731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:39.058817 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:39.059392 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:39.059404 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:39.061432 (+ 2028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":87,"threads_started":1}
W20260924 16:02:39.061698 2857 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:39.055887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:33035 (local address 127.25.109.62:39381)
0924 16:02:39.056049 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:39.056055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:39.056200 (+ 145us) server_negotiation.cc:407] Connection header received
0924 16:02:39.056369 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:39.056373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:39.056445 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:39.056547 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:39.056988 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:39.057516 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:39.058321 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:39.058558 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:39.058714 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:39.061577 (+ 2863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.3:33035: BlockingRecv error: recv got EOF from 127.25.109.3:33035 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 24 16:02:39 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:39 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:39.673368 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:39.667687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.1:44183)
0924 16:02:39.667990 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:39.668005 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:39.668093 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:39.668321 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:39.668323 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:39.668607 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:39.668745 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:39.668752 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:39.669485 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:39.669487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:39.670051 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:39.670058 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:39.670482 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:39.670485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:39.670549 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:39.671025 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:39.671036 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:39.673196 (+ 2160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":91,"threads_started":1}
W20260924 16:02:39.673715 2861 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:39.667786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:44183 (local address 127.25.109.62:39381)
0924 16:02:39.668078 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:39.668082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:39.668095 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:02:39.668164 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:39.668166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:39.668208 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:39.668283 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:39.668853 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:39.669395 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:39.670143 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:39.670361 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:39.670483 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:39.673364 (+ 2881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.1:44183: BlockingRecv error: recv got EOF from 127.25.109.1:44183 (error 108)
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":106,"threads_started":1}
Sep 24 16:02:39 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:39 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:39.870702 2862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:39.864904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.2:36317)
0924 16:02:39.865244 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:39.865261 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:39.865401 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:39.865689 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:39.865692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:39.865889 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:39.865984 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:39.865990 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:39.866808 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:39.866814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:39.867490 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:39.867504 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:39.868103 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:39.868107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:39.868207 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:39.868690 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:39.868701 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:39.870500 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":131,"threads_started":1}
W20260924 16:02:39.870770 2861 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:39.865080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:36317 (local address 127.25.109.62:39381)
0924 16:02:39.865246 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:39.865252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:39.865343 (+ 91us) server_negotiation.cc:407] Connection header received
0924 16:02:39.865504 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:39.865507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:39.865561 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:39.865642 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:39.866104 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:39.866689 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:39.867662 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:39.867964 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:39.868089 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:39.870661 (+ 2572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.2:36317: BlockingRecv error: recv got EOF from 127.25.109.2:36317 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 24 16:02:40 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:40 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:40.067909 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:40.062483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.3:41415)
0924 16:02:40.062753 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:40.062768 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:40.062883 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:40.063181 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:40.063184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:40.063433 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:40.063561 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:40.063569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:40.064449 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:40.064453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:40.065042 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:40.065050 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:40.065495 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:40.065497 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:40.065558 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:40.066056 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:40.066067 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:40.067795 (+ 1728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":98,"threads_started":1}
W20260924 16:02:40.068100 2861 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:40.062518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:41415 (local address 127.25.109.62:39381)
0924 16:02:40.062659 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:40.062666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:40.062865 (+ 199us) server_negotiation.cc:407] Connection header received
0924 16:02:40.062983 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:40.062987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:40.063052 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:40.063144 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:40.063684 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:40.064320 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:40.065145 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:40.065361 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:40.065473 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:40.067938 (+ 2465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.3:41415: BlockingRecv error: recv got EOF from 127.25.109.3:41415 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 16:02:40 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:40 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:40.679549 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:40.674215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.1:46341)
0924 16:02:40.674488 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:40.674502 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:40.674597 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:40.674900 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:40.674902 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:40.675082 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:40.675188 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:40.675194 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:40.675917 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:40.675919 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:40.676502 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:40.676510 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:40.676954 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:40.676956 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:40.677017 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:40.677569 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:40.677582 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:40.679434 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":86,"threads_started":1}
W20260924 16:02:40.679757 2865 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:40.674265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:46341 (local address 127.25.109.62:39381)
0924 16:02:40.674521 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:40.674534 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:40.674548 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:02:40.674733 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:40.674736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:40.674785 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:40.674867 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:40.675294 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:40.675826 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:40.676622 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:40.676851 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:40.676939 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:40.679614 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.1:46341: BlockingRecv error: recv got EOF from 127.25.109.1:46341 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":85,"threads_started":1}
Sep 24 16:02:40 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:40 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:40.877053 2866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:40.871523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.2:46141)
0924 16:02:40.871854 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:40.871868 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:40.871970 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:40.872207 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:40.872209 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:40.872392 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:40.872513 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:40.872519 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:40.873336 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:40.873339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:40.873916 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:40.873925 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:40.874362 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:40.874365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:40.874429 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:40.875003 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:40.875016 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:40.876933 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":130,"threads_started":1}
W20260924 16:02:40.877262 2865 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:40.871591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:46141 (local address 127.25.109.62:39381)
0924 16:02:40.871774 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:40.871782 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:40.871944 (+ 162us) server_negotiation.cc:407] Connection header received
0924 16:02:40.872053 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:40.872055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:40.872104 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:40.872173 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:40.872629 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:40.873233 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:40.874026 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:40.874261 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:40.874344 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:40.877125 (+ 2781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.2:46141: BlockingRecv error: recv got EOF from 127.25.109.2:46141 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 24 16:02:41 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:41 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:41.074404 2867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:41.068673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.3:39111)
0924 16:02:41.068960 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:41.068974 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:41.069073 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:41.069300 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:41.069302 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:41.069477 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:41.069570 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:41.069576 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:41.070394 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:41.070397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:41.071038 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:41.071048 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:41.071570 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:41.071573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:41.071674 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:41.072240 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:41.072252 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:41.074231 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":118,"threads_started":1}
W20260924 16:02:41.074469 2865 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:41.068741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:39111 (local address 127.25.109.62:39381)
0924 16:02:41.068896 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:41.068901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:41.069038 (+ 137us) server_negotiation.cc:407] Connection header received
0924 16:02:41.069151 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:41.069153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:41.069200 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:41.069267 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:41.069691 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:41.070297 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:41.071157 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:41.071463 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:41.071556 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:41.074362 (+ 2806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.3:39111: BlockingRecv error: recv got EOF from 127.25.109.3:39111 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 16:02:41 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:41 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:41.685730 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:41.680325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.1:42261)
0924 16:02:41.680673 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:41.680686 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:41.680782 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:41.681004 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:41.681006 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:41.681185 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:41.681277 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:41.681283 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:41.682040 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:41.682042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:41.682606 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:41.682614 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:41.683107 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:41.683110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:41.683169 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:41.683758 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:41.683769 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:41.685614 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":115,"threads_started":1}
W20260924 16:02:41.685920 2869 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:41.680388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:42261 (local address 127.25.109.62:39381)
0924 16:02:41.680703 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:41.680707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:41.680735 (+ 28us) server_negotiation.cc:407] Connection header received
0924 16:02:41.680868 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:41.680870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:41.680918 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:41.681001 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:41.681372 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:41.681937 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:41.682739 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:41.683002 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:41.683096 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:41.685749 (+ 2653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.1:42261: BlockingRecv error: recv got EOF from 127.25.109.1:42261 (error 108)
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":64,"threads_started":1}
Sep 24 16:02:41 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:41 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:41.883445 2870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:41.877847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.2:59649)
0924 16:02:41.878140 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:41.878156 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:41.878254 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:41.878479 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:41.878481 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:41.878679 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:41.878779 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:41.878785 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:41.879614 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:41.879626 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:41.880173 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:41.880182 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:41.880685 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:41.880688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:41.880751 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:41.881320 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:41.881332 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:41.883251 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":86,"threads_started":1}
W20260924 16:02:41.883507 2869 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:41.877887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:59649 (local address 127.25.109.62:39381)
0924 16:02:41.878073 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:41.878093 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:41.878221 (+ 128us) server_negotiation.cc:407] Connection header received
0924 16:02:41.878330 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:41.878332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:41.878380 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:41.878449 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:41.878883 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:41.879503 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:41.880325 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:41.880578 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:41.880673 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:41.883380 (+ 2707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.2:59649: BlockingRecv error: recv got EOF from 127.25.109.2:59649 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 24 16:02:42 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:42 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:42.087241 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:42.075109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.3:33289)
0924 16:02:42.075431 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:42.075445 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:42.075548 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:42.075798 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:42.075801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:42.075982 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:42.076084 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:42.076089 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.076825 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.076828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:42.077387 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:42.077395 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.077822 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.077825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:42.077891 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:42.084890 (+ 6999us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:42.084909 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:42.087120 (+ 2211us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":78,"threads_started":1}
W20260924 16:02:42.087431 2869 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:42.075146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:33289 (local address 127.25.109.62:39381)
0924 16:02:42.075297 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:42.075301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:42.075511 (+ 210us) server_negotiation.cc:407] Connection header received
0924 16:02:42.075643 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:42.075645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:42.075694 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:42.075759 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:42.076176 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.076730 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.077483 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.077719 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.077794 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:42.087267 (+ 9473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.3:33289: BlockingRecv error: recv got EOF from 127.25.109.3:33289 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 16:02:42 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:42 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:42.691745 2872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:42.686445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.1:38803)
0924 16:02:42.686746 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:42.686763 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:42.686876 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:42.687095 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:42.687097 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:42.687283 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:42.687380 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:42.687385 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.688130 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.688133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:42.688711 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:42.688719 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.689173 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.689176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:42.689237 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:42.689720 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:42.689732 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:42.691573 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":72,"threads_started":1}
W20260924 16:02:42.691821 2873 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:42.686615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:38803 (local address 127.25.109.62:39381)
0924 16:02:42.686886 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:42.686889 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:42.686902 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:02:42.686938 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:42.686940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:42.686981 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:42.687056 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:42.687507 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.688029 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.688838 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.689063 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.689152 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:42.691692 (+ 2540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.1:38803: BlockingRecv error: recv got EOF from 127.25.109.1:38803 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":88,"threads_started":1}
Sep 24 16:02:42 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:42 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:42.889892 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:42.884286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.2:35439)
0924 16:02:42.884566 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:42.884587 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:42.884725 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:42.884997 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:42.885000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:42.885212 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:42.885305 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:42.885311 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.886223 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.886228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:42.886805 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:42.886813 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.887268 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.887272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:42.887356 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:42.887849 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:42.887861 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:42.889767 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":108,"threads_started":1}
W20260924 16:02:42.890072 2873 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:42.884359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:35439 (local address 127.25.109.62:39381)
0924 16:02:42.884534 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:42.884538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:42.884664 (+ 126us) server_negotiation.cc:407] Connection header received
0924 16:02:42.884825 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:42.884828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:42.884883 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:42.884970 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:42.885425 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.886113 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.886925 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:42.887155 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:42.887264 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:42.889939 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.2:35439: BlockingRecv error: recv got EOF from 127.25.109.2:35439 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 16:02:43 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:43 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:43.093703 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:43.088003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.3:34309)
0924 16:02:43.088346 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:43.088362 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:43.088513 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:43.088814 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:43.088818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:43.089067 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:43.089186 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:43.089193 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:43.089965 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:43.089970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:43.090529 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:43.090537 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:43.090981 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:43.090985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:43.091070 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:43.091665 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:43.091677 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:43.093588 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":123,"threads_started":1}
W20260924 16:02:43.093850 2873 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:43.088161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:34309 (local address 127.25.109.62:39381)
0924 16:02:43.088320 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:43.088325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:43.088438 (+ 113us) server_negotiation.cc:407] Connection header received
0924 16:02:43.088632 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:43.088635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:43.088687 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:43.088788 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:43.089299 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:43.089854 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:43.090648 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:43.090870 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:43.090978 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:43.093733 (+ 2755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.3:34309: BlockingRecv error: recv got EOF from 127.25.109.3:34309 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 16:02:43 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:43 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:43.698438 2876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:43.692654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.1:60491)
0924 16:02:43.693045 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:43.693064 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:43.693167 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:43.693467 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:43.693469 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:43.693745 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:43.693871 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:43.693878 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:43.694651 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:43.694654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:43.695226 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:43.695235 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:43.695641 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:43.695644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:43.695707 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:43.696283 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:43.696296 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:43.698263 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":102,"threads_started":1}
W20260924 16:02:43.698513 2877 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:43.692757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:60491 (local address 127.25.109.62:39381)
0924 16:02:43.693009 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:43.693013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:43.693157 (+ 144us) server_negotiation.cc:407] Connection header received
0924 16:02:43.693246 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:43.693249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:43.693311 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:43.693419 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:43.693993 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:43.694528 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:43.695328 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:43.695544 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:43.695616 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:43.698389 (+ 2773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.1:60491: BlockingRecv error: recv got EOF from 127.25.109.1:60491 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":82,"threads_started":1}
Sep 24 16:02:43 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:43 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:43.896546 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:43.890807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.2:40039)
0924 16:02:43.891122 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:43.891141 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:43.891270 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:43.891649 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:43.891652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:43.891912 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:43.892054 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:43.892062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:43.892827 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:43.892829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:43.893416 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:43.893424 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:43.893841 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:43.893843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:43.893914 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:43.894424 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:43.894436 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:43.896325 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":134,"threads_started":1}
W20260924 16:02:43.896595 2877 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:43.890886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:40039 (local address 127.25.109.62:39381)
0924 16:02:43.891035 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:43.891041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:43.891225 (+ 184us) server_negotiation.cc:407] Connection header received
0924 16:02:43.891408 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:43.891415 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:43.891478 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:43.891551 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:43.892182 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:43.892730 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:43.893519 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:43.893739 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:43.893814 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:43.896490 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.2:40039: BlockingRecv error: recv got EOF from 127.25.109.2:40039 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 16:02:44 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:44 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:44.100687 2879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:44.094617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.3:41859)
0924 16:02:44.094952 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:44.094966 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:44.095060 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:44.095324 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:44.095325 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:44.095505 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:44.095615 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:44.095620 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:44.096339 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:44.096341 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:44.097000 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:44.097011 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:44.097761 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:44.097766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:44.097864 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:44.098532 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:44.098546 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:44.100513 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":161,"threads_started":1}
W20260924 16:02:44.100771 2877 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:44.094867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:41859 (local address 127.25.109.62:39381)
0924 16:02:44.095028 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:44.095033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:44.095046 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:02:44.095155 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:44.095157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:44.095206 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:44.095282 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:44.095715 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:44.096247 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:44.097258 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:44.097617 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:44.097754 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:44.100655 (+ 2901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.3:41859: BlockingRecv error: recv got EOF from 127.25.109.3:41859 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 24 16:02:44 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:44 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:44.704854 2880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:44.699375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.1:45267)
0924 16:02:44.699667 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:44.699680 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:44.699765 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:44.699988 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:44.699990 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:44.700180 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:44.700276 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:44.700281 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:44.701102 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:44.701105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:44.701664 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:44.701671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:44.702110 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:44.702113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:44.702184 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:44.702716 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:44.702728 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:44.704697 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":114,"threads_started":1}
W20260924 16:02:44.704974 2881 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:44.699430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:45267 (local address 127.25.109.62:39381)
0924 16:02:44.699705 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:44.699709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:44.699722 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:02:44.699849 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:44.699852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:44.699897 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:44.699978 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:44.700388 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:44.700999 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:44.701775 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:44.702006 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:44.702090 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:44.704803 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.1:45267: BlockingRecv error: recv got EOF from 127.25.109.1:45267 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":108,"threads_started":1}
Sep 24 16:02:44 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:44 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:44.902769 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:44.897458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.2:55589)
0924 16:02:44.897721 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:44.897733 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:44.897829 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:44.898075 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:44.898079 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:44.898257 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:44.898353 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:44.898358 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:44.899136 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:44.899139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:44.899685 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:44.899692 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:44.900139 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:44.900141 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:44.900202 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:44.900714 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:44.900726 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:44.902594 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":87,"threads_started":1}
W20260924 16:02:44.902912 2881 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:44.897515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:55589 (local address 127.25.109.62:39381)
0924 16:02:44.897670 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:44.897673 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:44.897801 (+ 128us) server_negotiation.cc:407] Connection header received
0924 16:02:44.897908 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:44.897911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:44.897954 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:44.898037 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:44.898486 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:44.899037 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:44.899810 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:44.900040 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:44.900122 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:44.902761 (+ 2639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.2:55589: BlockingRecv error: recv got EOF from 127.25.109.2:55589 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 16:02:45 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:45 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:45.106834 2883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:45.101580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.3:50489)
0924 16:02:45.101882 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:45.101895 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:45.101987 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:45.102281 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:45.102283 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:45.102466 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:45.102562 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:45.102568 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.103343 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.103346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:45.103902 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:45.103909 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.104358 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.104361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:45.104447 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:45.104915 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:45.104926 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:45.106715 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":90,"threads_started":1}
W20260924 16:02:45.107079 2881 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:45.101689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:50489 (local address 127.25.109.62:39381)
0924 16:02:45.101841 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:45.101844 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:45.101996 (+ 152us) server_negotiation.cc:407] Connection header received
0924 16:02:45.102119 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:45.102121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:45.102171 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:45.102244 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:45.102700 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.103246 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.104031 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.104258 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.104341 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:45.106852 (+ 2511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.3:50489: BlockingRecv error: recv got EOF from 127.25.109.3:50489 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 24 16:02:45 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:45 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:45.711781 2884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:45.705938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.1:39647)
0924 16:02:45.706257 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:45.706271 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:45.706368 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:45.706675 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:45.706677 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:45.706862 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:45.706960 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:45.706966 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.707702 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.707704 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:45.708265 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:45.708274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.708785 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.708788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:45.708857 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:45.709340 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:45.709351 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:45.711607 (+ 2256us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":99,"threads_started":1}
W20260924 16:02:45.711880 2885 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:45.706053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:39647 (local address 127.25.109.62:39381)
0924 16:02:45.706346 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:45.706350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:45.706364 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:02:45.706505 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:45.706508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:45.706555 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:45.706636 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:45.707070 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.707595 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.708366 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.708676 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.708757 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:45.711758 (+ 3001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.1:39647: BlockingRecv error: recv got EOF from 127.25.109.1:39647 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":86,"threads_started":1}
Sep 24 16:02:45 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:45 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:45.909212 2886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:45.903742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.2:38133)
0924 16:02:45.904029 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:45.904044 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:45.904154 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:45.904445 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:45.904451 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:45.904663 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:45.904763 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:45.904768 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.905649 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.905653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:45.906402 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:45.906412 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.906820 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.906822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:45.906882 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:45.907349 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:45.907361 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:45.909095 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":96,"threads_started":1}
W20260924 16:02:45.909349 2885 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:45.903820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:38133 (local address 127.25.109.62:39381)
0924 16:02:45.904012 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:45.904016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:45.904125 (+ 109us) server_negotiation.cc:407] Connection header received
0924 16:02:45.904237 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:45.904240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:45.904285 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:45.904364 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:45.904923 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.905508 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.906504 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:45.906726 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:45.906808 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:45.909232 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.2:38133: BlockingRecv error: recv got EOF from 127.25.109.2:38133 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 24 16:02:46 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:46 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:46.113076 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:46.107644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.3:47521)
0924 16:02:46.107936 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:46.107949 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:46.108047 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:46.108341 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:46.108343 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:46.108546 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:46.108655 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:46.108661 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:46.109394 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:46.109397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:46.109948 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:46.109957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:46.110408 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:46.110411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:46.110467 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:46.110986 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:46.110997 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:46.112882 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":107,"threads_started":1}
W20260924 16:02:46.113102 2885 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:46.107721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:47521 (local address 127.25.109.62:39381)
0924 16:02:46.107868 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:46.107872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:46.108016 (+ 144us) server_negotiation.cc:407] Connection header received
0924 16:02:46.108171 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:46.108174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:46.108228 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:46.108306 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:46.108765 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:46.109300 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:46.110074 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:46.110303 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:46.110391 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:46.113007 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.3:47521: BlockingRecv error: recv got EOF from 127.25.109.3:47521 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 16:02:46 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:46 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:46.717732 2888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:46.712606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.1:40149)
0924 16:02:46.712872 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:46.712896 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:46.712991 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:46.713238 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:46.713241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:46.713419 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:46.713516 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:46.713521 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:46.714232 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:46.714235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:46.714773 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:46.714780 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:46.715188 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:46.715191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:46.715252 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:46.715767 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:46.715778 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:46.717609 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":71,"threads_started":1}
W20260924 16:02:46.717900 2889 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:46.712718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:40149 (local address 127.25.109.62:39381)
0924 16:02:46.712965 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:46.712968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:46.712982 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:02:46.713072 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:46.713075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:46.713118 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:46.713197 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:46.713616 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:46.714141 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:46.714871 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:46.715087 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:46.715163 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:46.717758 (+ 2595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.1:40149: BlockingRecv error: recv got EOF from 127.25.109.1:40149 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":77,"threads_started":1}
Sep 24 16:02:46 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:46 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:46.915705 2890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:46.910111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.2:42301)
0924 16:02:46.910439 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:46.910456 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:46.910605 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:46.910874 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:46.910877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:46.911103 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:46.911208 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:46.911213 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:46.911970 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:46.911975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:46.912571 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:46.912580 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:46.913028 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:46.913031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:46.913127 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:46.913693 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:46.913703 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:46.915520 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":90,"threads_started":1}
W20260924 16:02:46.915772 2889 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:46.910248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:42301 (local address 127.25.109.62:39381)
0924 16:02:46.910422 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:46.910427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:46.910540 (+ 113us) server_negotiation.cc:407] Connection header received
0924 16:02:46.910706 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:46.910709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:46.910763 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:46.910853 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:46.911329 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:46.911861 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:46.912694 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:46.912915 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:46.913024 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:46.915657 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.2:42301: BlockingRecv error: recv got EOF from 127.25.109.2:42301 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 24 16:02:47 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:47 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:47.122771 2891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:47.113762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.3:57031)
0924 16:02:47.114060 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:47.114073 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:47.114169 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:47.114409 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:47.114411 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:47.114586 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:47.114692 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:47.114697 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:47.115398 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:47.115401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:47.115947 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:47.115954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:47.116354 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:47.116357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:47.116434 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:47.120638 (+ 4204us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:47.120650 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:47.122587 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":79,"threads_started":1}
W20260924 16:02:47.122825 2889 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:47.113862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:57031 (local address 127.25.109.62:39381)
0924 16:02:47.113990 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:47.113994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:47.114136 (+ 142us) server_negotiation.cc:407] Connection header received
0924 16:02:47.114249 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:47.114251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:47.114297 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:47.114372 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:47.114787 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:47.115309 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:47.116043 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:47.116260 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:47.116334 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:47.122717 (+ 6383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.3:57031: BlockingRecv error: recv got EOF from 127.25.109.3:57031 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 16:02:47 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:47 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:47.724660 2892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:47.718652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.1:42275)
0924 16:02:47.718934 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:47.718948 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:47.719041 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:47.719298 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:47.719300 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:47.719485 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:47.719584 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:47.719590 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:47.720351 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:47.720354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:47.721021 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:47.721032 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:47.721588 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:47.721591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:47.721664 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:47.722214 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:47.722226 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:47.724456 (+ 2230us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":93,"threads_started":1}
W20260924 16:02:47.724720 2893 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:47.718716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:42275 (local address 127.25.109.62:39381)
0924 16:02:47.718988 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:47.718992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:47.719005 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:02:47.719128 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:47.719131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:47.719172 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:47.719250 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:47.719689 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:47.720235 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:47.721219 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:47.721450 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:47.721541 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:47.724587 (+ 3046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.1:42275: BlockingRecv error: recv got EOF from 127.25.109.1:42275 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":107,"threads_started":1}
Sep 24 16:02:47 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:47 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:47.922462 2894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:47.916646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.2:42707)
0924 16:02:47.916936 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:47.916951 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:47.917093 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:47.917369 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:47.917372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:47.917584 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:47.917682 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:47.917688 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:47.918467 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:47.918472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:47.919043 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:47.919051 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:47.919534 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:47.919538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:47.919620 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:47.920232 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:47.920244 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:47.922288 (+ 2044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":101,"threads_started":1}
W20260924 16:02:47.922529 2893 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:47.916945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:42707 (local address 127.25.109.62:39381)
0924 16:02:47.917097 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:47.917101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:47.917117 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:02:47.917189 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:47.917192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:47.917240 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:47.917336 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:47.917810 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:47.918350 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:47.919171 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:47.919413 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:47.919538 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:47.922418 (+ 2880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.2:42707: BlockingRecv error: recv got EOF from 127.25.109.2:42707 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 16:02:48 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:48 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:48.129362 2895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:48.123581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.3:55011)
0924 16:02:48.123919 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:48.123940 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:48.124091 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:48.124381 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:48.124384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:48.124662 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:48.124761 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:48.124767 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:48.125564 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:48.125569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:48.126136 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:48.126145 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:48.126632 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:48.126636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:48.126726 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:48.127229 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:48.127242 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:48.129177 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":84,"threads_started":1}
W20260924 16:02:48.129472 2893 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:48.123692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:55011 (local address 127.25.109.62:39381)
0924 16:02:48.123873 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:48.123879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:48.124024 (+ 145us) server_negotiation.cc:407] Connection header received
0924 16:02:48.124195 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:48.124199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:48.124260 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:48.124355 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:48.124895 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:48.125444 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:48.126265 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:48.126508 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:48.126624 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:48.129346 (+ 2722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.3:55011: BlockingRecv error: recv got EOF from 127.25.109.3:55011 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 24 16:02:48 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:48 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:48.731676 2896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:48.725686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.1:50763)
0924 16:02:48.725991 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:48.726007 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:48.726122 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:48.726377 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:48.726378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:48.726561 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:48.726670 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:48.726675 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:48.727735 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:48.727740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:48.728332 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:48.728341 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:48.728892 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:48.728896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:48.728986 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:48.729471 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:48.729482 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:48.731487 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":77,"threads_started":1}
W20260924 16:02:48.731814 2897 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:48.725767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:50763 (local address 127.25.109.62:39381)
0924 16:02:48.726151 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:48.726155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:48.726185 (+ 30us) server_negotiation.cc:407] Connection header received
0924 16:02:48.726224 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:48.726226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:48.726271 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:48.726342 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:48.726850 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:48.727632 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:48.728523 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:48.728769 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:48.728884 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:48.731638 (+ 2754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.1:50763: BlockingRecv error: recv got EOF from 127.25.109.1:50763 (error 108)
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":80,"threads_started":1}
Sep 24 16:02:48 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:48 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:48.928956 2898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:48.923537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.2:33353)
0924 16:02:48.923823 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:48.923836 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:48.923935 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:48.924163 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:48.924165 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:48.924347 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:48.924473 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:48.924479 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:48.925168 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:48.925171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:48.925763 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:48.925773 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:48.926275 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:48.926278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:48.926372 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:48.926947 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:48.926959 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:48.928782 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":91,"threads_started":1}
W20260924 16:02:48.929024 2897 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:48.923581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:33353 (local address 127.25.109.62:39381)
0924 16:02:48.923723 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:48.923727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:48.923901 (+ 174us) server_negotiation.cc:407] Connection header received
0924 16:02:48.924003 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:48.924005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:48.924049 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:48.924122 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:48.924556 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:48.925073 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:48.925925 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:48.926150 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:48.926268 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:48.928909 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.2:33353: BlockingRecv error: recv got EOF from 127.25.109.2:33353 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 16:02:49 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:49 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.3@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:49.136016 2899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:49.130308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.3:47851)
0924 16:02:49.130658 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:49.130672 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:49.130786 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:49.131043 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:49.131046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:49.131260 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:49.131377 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:49.131383 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:49.132221 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:49.132224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:49.132827 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:49.132835 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:49.133266 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:49.133269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:49.133341 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:49.133861 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:49.133872 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:49.135863 (+ 1991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":91,"threads_started":1}
W20260924 16:02:49.136154 2897 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:49.130381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:47851 (local address 127.25.109.62:39381)
0924 16:02:49.130555 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:49.130565 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:49.130755 (+ 190us) server_negotiation.cc:407] Connection header received
0924 16:02:49.130883 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:49.130886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:49.130938 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:49.131019 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:49.131482 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:49.132117 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:49.132933 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:49.133165 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:49.133246 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:49.136032 (+ 2786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.3:47851: BlockingRecv error: recv got EOF from 127.25.109.3:47851 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 24 16:02:49 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:49 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.1@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:49.737937 2900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:49.732638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.1:32899)
0924 16:02:49.732919 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:49.732934 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:49.733020 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:49.733261 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:49.733263 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:49.733446 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:49.733540 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:49.733545 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:49.734309 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:49.734311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:49.734866 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:49.734873 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:49.735336 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:49.735338 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:49.735401 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:49.735910 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:49.735921 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:49.737815 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":100,"threads_started":1}
W20260924 16:02:49.738160 2901 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:49.732771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:32899 (local address 127.25.109.62:39381)
0924 16:02:49.733006 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:49.733010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:49.733025 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:02:49.733102 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:49.733105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:49.733146 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:49.733219 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:49.733663 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:49.734211 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:49.734985 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:49.735217 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:49.735315 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:49.738001 (+ 2686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.1:32899: BlockingRecv error: recv got EOF from 127.25.109.1:32899 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":84,"threads_started":1}
Sep 24 16:02:49 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:02:49 dist-test-slave-n106 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.109.2@KRBTEST.COM for oryx/127.25.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:02:49.935606 2902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:49.930010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39381 (local address 127.25.109.2:33745)
0924 16:02:49.930334 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:49.930368 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:49.930483 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:49.930793 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:49.930796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:49.931042 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:49.931175 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:49.931183 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:49.932051 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:49.932053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:49.932640 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:49.932648 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:49.933045 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:49.933048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:49.933110 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:49.933633 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:49.933644 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:49.935476 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.109.62:39381: Server oryx/127.25.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":108,"threads_started":1}
W20260924 16:02:49.935758 2901 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:49.930165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:33745 (local address 127.25.109.62:39381)
0924 16:02:49.930325 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:49.930330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:49.930469 (+ 139us) server_negotiation.cc:407] Connection header received
0924 16:02:49.930604 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:49.930608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:49.930670 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:49.930755 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:49.931288 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:49.931948 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:49.932737 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:49.932950 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:49.933031 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:49.935602 (+ 2571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.109.2:33745: BlockingRecv error: recv got EOF from 127.25.109.2:33745 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 16:02:50.010859 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 2420
I20260924 16:02:50.017076 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 2575
I20260924 16:02:50.022805 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 2707
I20260924 16:02:50.028538 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 2349
2026-09-24T16:02:50Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18024 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T16:02:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:02:50Z Disabled control of system clock
I20260924 16:02:50.058403 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:38591
--webserver_interface=127.25.109.62
--webserver_port=0
--builtin_ntp_servers=127.25.109.20:46419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:38591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 16:02:50.167891 2906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:50.168156 2906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:50.168190 2906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:50.172125 2906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:02:50.172174 2906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:50.172190 2906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:50.172207 2906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:02:50.172271 2906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:02:50.176920 2906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:46419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:38591
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:38591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:50.177949 2906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:50.179311 2906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:50.187608 2912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:50.187716 2911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:50.187772 2906 server_base.cc:1034] running on GCE node
W20260924 16:02:50.187603 2914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:50.188570 2906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:50.189661 2906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:50.190829 2906 hybrid_clock.cc:648] HybridClock initialized: now 1790265770190800 us; error 42 us; skew 500 ppm
I20260924 16:02:50.192791 2906 webserver.cc:460] Webserver started at http://127.25.109.62:35785/ using document root <none> and password file <none>
I20260924 16:02:50.193337 2906 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:50.193394 2906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:50.193564 2906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:50.195592 2906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "4de9d83ebadc400988d4aa07786f43c0"
format_stamp: "Formatted at 2026-09-24 16:02:50 on dist-test-slave-n106"
server_key: "25de638769eaa10f098b169fcec4dd50"
server_key_iv: "ac451bb98b20e53a7b8ea2d52bd94ac9"
server_key_version: "encryptionkey@0"
I20260924 16:02:50.196106 2906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "4de9d83ebadc400988d4aa07786f43c0"
format_stamp: "Formatted at 2026-09-24 16:02:50 on dist-test-slave-n106"
server_key: "25de638769eaa10f098b169fcec4dd50"
server_key_iv: "ac451bb98b20e53a7b8ea2d52bd94ac9"
server_key_version: "encryptionkey@0"
I20260924 16:02:50.199661 2906 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 16:02:50.202142 2920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:50.203358 2906 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:02:50.203477 2906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "4de9d83ebadc400988d4aa07786f43c0"
format_stamp: "Formatted at 2026-09-24 16:02:50 on dist-test-slave-n106"
server_key: "25de638769eaa10f098b169fcec4dd50"
server_key_iv: "ac451bb98b20e53a7b8ea2d52bd94ac9"
server_key_version: "encryptionkey@0"
I20260924 16:02:50.203572 2906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:50.218715 2906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:50.219297 2906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:50.219439 2906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:50.226156 2906 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:38591
I20260924 16:02:50.226172 2972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:38591 every 8 connection(s)
I20260924 16:02:50.227116 2906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:02:50.230201 2973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:50.233916 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 2906
I20260924 16:02:50.234020 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:02:50.234331 26036 external_mini_cluster.cc:1428] Setting key 0ff449ad43c08b2523a13cb5e4eef77a
I20260924 16:02:50.235991 2973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de9d83ebadc400988d4aa07786f43c0: Bootstrap starting.
I20260924 16:02:50.238782 2973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4de9d83ebadc400988d4aa07786f43c0: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:50.239668 2973 log.cc:824] T 00000000000000000000000000000000 P 4de9d83ebadc400988d4aa07786f43c0: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:50.241758 2973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de9d83ebadc400988d4aa07786f43c0: No bootstrap required, opened a new log
I20260924 16:02:50.245226 2976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:50.235828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45116 (local address 127.25.109.62:38591)
0924 16:02:50.236535 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:50.236555 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:50.236597 (+ 42us) server_negotiation.cc:407] Connection header received
0924 16:02:50.237477 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:50.237502 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:50.238001 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:50.238367 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:50.238831 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:50.240255 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:50.241229 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:50.241909 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:50.242065 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:50.242996 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:50.243002 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:50.243017 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:02:50.243537 (+ 520us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:50.243605 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:02:50.243633 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:50.243735 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:50.243897 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 16:02:50.244166 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":105,"threads_started":1}
I20260924 16:02:50.246229 2973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4de9d83ebadc400988d4aa07786f43c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de9d83ebadc400988d4aa07786f43c0" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 38591 } }
I20260924 16:02:50.246465 2973 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4de9d83ebadc400988d4aa07786f43c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:50.246526 2973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4de9d83ebadc400988d4aa07786f43c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4de9d83ebadc400988d4aa07786f43c0, State: Initialized, Role: FOLLOWER
I20260924 16:02:50.246982 2973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4de9d83ebadc400988d4aa07786f43c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de9d83ebadc400988d4aa07786f43c0" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 38591 } }
I20260924 16:02:50.247102 2973 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4de9d83ebadc400988d4aa07786f43c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:02:50.247182 2973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4de9d83ebadc400988d4aa07786f43c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:02:50.247299 2973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4de9d83ebadc400988d4aa07786f43c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:50.248281 2973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4de9d83ebadc400988d4aa07786f43c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de9d83ebadc400988d4aa07786f43c0" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 38591 } }
I20260924 16:02:50.248646 2973 leader_election.cc:302] T 00000000000000000000000000000000 P 4de9d83ebadc400988d4aa07786f43c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4de9d83ebadc400988d4aa07786f43c0; no voters:
I20260924 16:02:50.248934 2973 leader_election.cc:288] T 00000000000000000000000000000000 P 4de9d83ebadc400988d4aa07786f43c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:02:50.249037 2978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4de9d83ebadc400988d4aa07786f43c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:50.249302 2978 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4de9d83ebadc400988d4aa07786f43c0 [term 1 LEADER]: Becoming Leader. State: Replica: 4de9d83ebadc400988d4aa07786f43c0, State: Running, Role: LEADER
I20260924 16:02:50.249673 2978 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4de9d83ebadc400988d4aa07786f43c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de9d83ebadc400988d4aa07786f43c0" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 38591 } }
I20260924 16:02:50.250167 2973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4de9d83ebadc400988d4aa07786f43c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:02:50.251359 2979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de9d83ebadc400988d4aa07786f43c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4de9d83ebadc400988d4aa07786f43c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de9d83ebadc400988d4aa07786f43c0" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 38591 } } }
I20260924 16:02:50.251363 2980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de9d83ebadc400988d4aa07786f43c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4de9d83ebadc400988d4aa07786f43c0. Latest consensus state: current_term: 1 leader_uuid: "4de9d83ebadc400988d4aa07786f43c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de9d83ebadc400988d4aa07786f43c0" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 38591 } } }
I20260924 16:02:50.251513 2979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de9d83ebadc400988d4aa07786f43c0 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:50.251513 2980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de9d83ebadc400988d4aa07786f43c0 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:50.251955 2987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:02:50.254892 2987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:02:50.260283 2987 catalog_manager.cc:1347] Generated new cluster ID: 8605732f8c8a4adab283ec9f247cbd5b
I20260924 16:02:50.260344 2987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:02:50.271670 2987 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:02:50.272830 2987 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:02:50.279222 2987 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4de9d83ebadc400988d4aa07786f43c0: Generated new TSK 0
I20260924 16:02:50.279907 2987 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:02:50.286468 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:0
--local_ip_for_outbound_sockets=127.25.109.1
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:38591
--builtin_ntp_servers=127.25.109.20:46419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 16:02:50.396948 2997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:50.397192 2997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:50.397233 2997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:50.401155 2997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:50.401202 2997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:50.401254 2997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:02:50.405933 2997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:46419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:38591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:50.406958 2997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:50.408380 2997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:50.417821 3005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:50.417830 3003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:50.417821 3002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:50.417989 2997 server_base.cc:1034] running on GCE node
I20260924 16:02:50.418511 2997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:50.419181 2997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:50.420385 2997 hybrid_clock.cc:648] HybridClock initialized: now 1790265770420347 us; error 44 us; skew 500 ppm
I20260924 16:02:50.423040 2997 webserver.cc:460] Webserver started at http://127.25.109.1:42991/ using document root <none> and password file <none>
I20260924 16:02:50.423837 2997 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:50.423936 2997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:50.424214 2997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:50.426992 2997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/data/instance:
uuid: "89083e324dcd4c589cab7278d4ba0927"
format_stamp: "Formatted at 2026-09-24 16:02:50 on dist-test-slave-n106"
server_key: "4810448602caa64494bcc8e364cf85f2"
server_key_iv: "e98d25199c7d3eab20c0e2c874fc8d15"
server_key_version: "encryptionkey@0"
I20260924 16:02:50.427750 2997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance:
uuid: "89083e324dcd4c589cab7278d4ba0927"
format_stamp: "Formatted at 2026-09-24 16:02:50 on dist-test-slave-n106"
server_key: "4810448602caa64494bcc8e364cf85f2"
server_key_iv: "e98d25199c7d3eab20c0e2c874fc8d15"
server_key_version: "encryptionkey@0"
I20260924 16:02:50.432344 2997 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:02:50.435036 3011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:50.436329 2997 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 16:02:50.436513 2997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "89083e324dcd4c589cab7278d4ba0927"
format_stamp: "Formatted at 2026-09-24 16:02:50 on dist-test-slave-n106"
server_key: "4810448602caa64494bcc8e364cf85f2"
server_key_iv: "e98d25199c7d3eab20c0e2c874fc8d15"
server_key_version: "encryptionkey@0"
I20260924 16:02:50.436623 2997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:50.453488 2997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:50.454125 2997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:50.454273 2997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:50.454802 2997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:50.455695 2997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:50.455744 2997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:50.455785 2997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:50.455806 2997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:50.465135 2997 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:44133
I20260924 16:02:50.465169 3125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:44133 every 8 connection(s)
I20260924 16:02:50.466136 2997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:02:50.472587 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 2997
I20260924 16:02:50.472725 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance
I20260924 16:02:50.473080 26036 external_mini_cluster.cc:1428] Setting key 623a6eac28e08c6ebe96e2c94ee5afd8
I20260924 16:02:50.475163 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:0
--local_ip_for_outbound_sockets=127.25.109.2
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:38591
--builtin_ntp_servers=127.25.109.20:46419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 16:02:50.476259 2976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:50.468298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:57159 (local address 127.25.109.62:38591)
0924 16:02:50.468490 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:50.468494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:50.469600 (+ 1106us) server_negotiation.cc:407] Connection header received
0924 16:02:50.471015 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:50.471018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:50.471076 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:50.471159 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:50.472062 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:50.472661 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:50.474226 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:50.474548 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:50.474683 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:50.475778 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:50.475782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:50.475784 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:02:50.475796 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:50.475813 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:02:50.475821 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:50.475896 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:50.476061 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 16:02:50.476121 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:02:50.477300 3128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:50.468757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:38591 (local address 127.25.109.1:57159)
0924 16:02:50.469429 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:50.469470 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:50.470793 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:50.471341 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:50.471348 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:50.471434 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:50.471915 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:50.471928 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:50.472833 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:50.472837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:50.474082 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:50.474093 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:50.474724 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:50.474728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:50.474899 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:50.475504 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:02:50.475506 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:50.475546 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:02:50.475611 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:50.475909 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:50.475921 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:50.475933 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 16:02:50.476059 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 16:02:50.476566 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":123,"threads_started":1}
I20260924 16:02:50.478668 3126 heartbeater.cc:347] Connected to a master server at 127.25.109.62:38591
I20260924 16:02:50.478909 3126 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:50.479423 3126 heartbeater.cc:511] Master 127.25.109.62:38591 requested a full tablet report, sending...
I20260924 16:02:50.480957 2937 ts_manager.cc:194] Registered new tserver with Master: 89083e324dcd4c589cab7278d4ba0927 (127.25.109.1:44133)
I20260924 16:02:50.482095 2937 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.1:57159
W20260924 16:02:50.589143 3129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:50.589448 3129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:50.589553 3129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:50.593554 3129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:50.593621 3129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:50.593700 3129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:02:50.598390 3129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:46419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:38591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:50.599397 3129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:50.600818 3129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:50.610388 3134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:50.610426 3137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:50.610538 3129 server_base.cc:1034] running on GCE node
W20260924 16:02:50.610404 3135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:50.611014 3129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:50.611632 3129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:50.612811 3129 hybrid_clock.cc:648] HybridClock initialized: now 1790265770612781 us; error 49 us; skew 500 ppm
I20260924 16:02:50.614646 3129 webserver.cc:460] Webserver started at http://127.25.109.2:34427/ using document root <none> and password file <none>
I20260924 16:02:50.615240 3129 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:50.615330 3129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:50.615559 3129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:50.617566 3129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/data/instance:
uuid: "334af8882ebc4972986fd9dbcd9ed999"
format_stamp: "Formatted at 2026-09-24 16:02:50 on dist-test-slave-n106"
server_key: "0462a530c3917544bffb65169cd96bc6"
server_key_iv: "cea0ce925f6317df2ec476ce1b543c5a"
server_key_version: "encryptionkey@0"
I20260924 16:02:50.618108 3129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance:
uuid: "334af8882ebc4972986fd9dbcd9ed999"
format_stamp: "Formatted at 2026-09-24 16:02:50 on dist-test-slave-n106"
server_key: "0462a530c3917544bffb65169cd96bc6"
server_key_iv: "cea0ce925f6317df2ec476ce1b543c5a"
server_key_version: "encryptionkey@0"
I20260924 16:02:50.621670 3129 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:02:50.624074 3143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:50.625264 3129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:02:50.625401 3129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "334af8882ebc4972986fd9dbcd9ed999"
format_stamp: "Formatted at 2026-09-24 16:02:50 on dist-test-slave-n106"
server_key: "0462a530c3917544bffb65169cd96bc6"
server_key_iv: "cea0ce925f6317df2ec476ce1b543c5a"
server_key_version: "encryptionkey@0"
I20260924 16:02:50.625509 3129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:50.634683 3129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:50.635303 3129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:50.635488 3129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:50.636025 3129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:50.637007 3129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:50.637077 3129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:50.637154 3129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:50.637207 3129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:50.645915 3129 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:38053
I20260924 16:02:50.645941 3257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:38053 every 8 connection(s)
I20260924 16:02:50.646839 3129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:02:50.651405 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 3129
I20260924 16:02:50.651522 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance
I20260924 16:02:50.651860 26036 external_mini_cluster.cc:1428] Setting key 2e488f1ae9bb5f6e95d14f3cb6f341ec
I20260924 16:02:50.653898 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:0
--local_ip_for_outbound_sockets=127.25.109.3
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:38591
--builtin_ntp_servers=127.25.109.20:46419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 16:02:50.656790 2976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:50.648781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:42981 (local address 127.25.109.62:38591)
0924 16:02:50.648947 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:50.648952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:50.650039 (+ 1087us) server_negotiation.cc:407] Connection header received
0924 16:02:50.651578 (+ 1539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:50.651582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:50.651652 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:50.651751 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:50.652873 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:50.653521 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:50.654768 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:50.655001 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:50.655122 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:50.656233 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:50.656236 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:50.656239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:02:50.656258 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:50.656279 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:02:50.656287 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:50.656362 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:50.656582 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 16:02:50.656647 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 16:02:50.657676 3260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:50.649195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:38591 (local address 127.25.109.2:42981)
0924 16:02:50.649849 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:50.649890 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:50.651338 (+ 1448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:50.651923 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:50.651935 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:50.652055 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:50.652722 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:50.652739 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:50.653647 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:50.653651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:50.654638 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:50.654648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:50.655162 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:50.655166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:50.655334 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:50.655967 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:02:50.655968 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:50.656010 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:02:50.656075 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:50.656377 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:50.656388 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:50.656439 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 16:02:50.656577 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 16:02:50.656856 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":109,"threads_started":1}
I20260924 16:02:50.658834 3258 heartbeater.cc:347] Connected to a master server at 127.25.109.62:38591
I20260924 16:02:50.659092 3258 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:50.659610 3258 heartbeater.cc:511] Master 127.25.109.62:38591 requested a full tablet report, sending...
I20260924 16:02:50.660874 2937 ts_manager.cc:194] Registered new tserver with Master: 334af8882ebc4972986fd9dbcd9ed999 (127.25.109.2:38053)
I20260924 16:02:50.661782 2937 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.2:42981
W20260924 16:02:50.766511 3261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:50.766791 3261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:50.766829 3261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:50.770974 3261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:50.771029 3261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:50.771085 3261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:02:50.775820 3261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:46419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:38591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:50.776955 3261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:50.778288 3261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:50.787842 3266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:50.787894 3269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:50.787926 3261 server_base.cc:1034] running on GCE node
W20260924 16:02:50.787850 3267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:50.788591 3261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:50.789151 3261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:50.790337 3261 hybrid_clock.cc:648] HybridClock initialized: now 1790265770790307 us; error 50 us; skew 500 ppm
I20260924 16:02:50.792064 3261 webserver.cc:460] Webserver started at http://127.25.109.3:38729/ using document root <none> and password file <none>
I20260924 16:02:50.792752 3261 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:50.792843 3261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:50.793073 3261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:50.795107 3261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/data/instance:
uuid: "8023d2e86d6140e4aa82dbf9926a2a29"
format_stamp: "Formatted at 2026-09-24 16:02:50 on dist-test-slave-n106"
server_key: "a0f98b573a2e2b0b65b922eb349a950a"
server_key_iv: "19e6066bc47f24972f8c231d40a702c8"
server_key_version: "encryptionkey@0"
I20260924 16:02:50.795683 3261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance:
uuid: "8023d2e86d6140e4aa82dbf9926a2a29"
format_stamp: "Formatted at 2026-09-24 16:02:50 on dist-test-slave-n106"
server_key: "a0f98b573a2e2b0b65b922eb349a950a"
server_key_iv: "19e6066bc47f24972f8c231d40a702c8"
server_key_version: "encryptionkey@0"
I20260924 16:02:50.799288 3261 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 16:02:50.801817 3275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:50.803009 3261 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:02:50.803143 3261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "8023d2e86d6140e4aa82dbf9926a2a29"
format_stamp: "Formatted at 2026-09-24 16:02:50 on dist-test-slave-n106"
server_key: "a0f98b573a2e2b0b65b922eb349a950a"
server_key_iv: "19e6066bc47f24972f8c231d40a702c8"
server_key_version: "encryptionkey@0"
I20260924 16:02:50.803256 3261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:50.820111 3261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:50.820853 3261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:50.821048 3261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:50.821621 3261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:50.822574 3261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:50.822649 3261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:50.822721 3261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:50.822772 3261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:50.831881 3261 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:46235
I20260924 16:02:50.831898 3388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:46235 every 8 connection(s)
I20260924 16:02:50.832887 3261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:02:50.840294 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 3261
I20260924 16:02:50.840413 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance
I20260924 16:02:50.840734 26036 external_mini_cluster.cc:1428] Setting key 8ad3a17d100401214f9308c11eb0bf20
I20260924 16:02:50.842579 2976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:50.834742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:55135 (local address 127.25.109.62:38591)
0924 16:02:50.834879 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:50.834883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:50.835991 (+ 1108us) server_negotiation.cc:407] Connection header received
0924 16:02:50.837508 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:50.837511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:50.837568 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:50.837681 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:50.838578 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:50.839122 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:50.840428 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:50.840699 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:50.840826 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:50.842088 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:50.842092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:50.842096 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:02:50.842109 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:50.842131 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:02:50.842142 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:50.842217 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:50.842415 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 16:02:50.842468 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 16:02:50.843582 3391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:50.835144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:38591 (local address 127.25.109.3:55135)
0924 16:02:50.835831 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:50.835871 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:50.837284 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:50.837854 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:50.837861 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:50.837945 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:50.838430 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:50.838443 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:50.839252 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:50.839258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:50.840278 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:50.840288 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:50.840868 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:50.840872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:50.841074 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:50.841828 (+ 754us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:02:50.841829 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:50.841869 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:02:50.841933 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:50.842254 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:50.842264 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:50.842272 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 16:02:50.842408 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 16:02:50.842690 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":108,"threads_started":1}
I20260924 16:02:50.844662 3389 heartbeater.cc:347] Connected to a master server at 127.25.109.62:38591
I20260924 16:02:50.844906 3389 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:50.845409 3389 heartbeater.cc:511] Master 127.25.109.62:38591 requested a full tablet report, sending...
I20260924 16:02:50.846451 2937 ts_manager.cc:194] Registered new tserver with Master: 8023d2e86d6140e4aa82dbf9926a2a29 (127.25.109.3:46235)
I20260924 16:02:50.846964 2937 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.3:55135
I20260924 16:02:50.854792 26036 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 16:02:50.857631 2976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:50.856727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45118 (local address 127.25.109.62:38591)
0924 16:02:50.856870 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:50.856874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:50.857060 (+ 186us) server_negotiation.cc:407] Connection header received
0924 16:02:50.857145 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:50.857147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:50.857187 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:50.857250 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:50.857503 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45118: BlockingRecv error: recv got EOF from 127.0.0.1:45118 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:02:50.857908 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 2997
I20260924 16:02:50.864801 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 3129
I20260924 16:02:50.870309 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 3261
I20260924 16:02:50.875566 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 2906
2026-09-24T16:02:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (845 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T16:02:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:02:50Z Disabled control of system clock
I20260924 16:02:50.904769 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:44077
--webserver_interface=127.25.109.62
--webserver_port=0
--builtin_ntp_servers=127.25.109.20:45039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:44077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 16:02:51.014868 3400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:51.015106 3400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:51.015143 3400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:51.019174 3400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:02:51.019227 3400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:51.019241 3400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:51.019258 3400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:02:51.019272 3400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:02:51.023964 3400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:45039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:44077
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:44077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:51.025009 3400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:51.026717 3400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:51.035161 3405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:51.035311 3400 server_base.cc:1034] running on GCE node
W20260924 16:02:51.035153 3408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:51.035168 3406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:51.035924 3400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:51.037084 3400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:51.038251 3400 hybrid_clock.cc:648] HybridClock initialized: now 1790265771038213 us; error 58 us; skew 500 ppm
I20260924 16:02:51.040138 3400 webserver.cc:460] Webserver started at http://127.25.109.62:46303/ using document root <none> and password file <none>
I20260924 16:02:51.040784 3400 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:51.040848 3400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:51.041085 3400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:51.043123 3400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "533bea1e6ba543a4a6e87e64dc2c39da"
format_stamp: "Formatted at 2026-09-24 16:02:51 on dist-test-slave-n106"
server_key: "72fc5ffc080cc821c47a7fd19398f4e3"
server_key_iv: "7480473f7ec14d116a4c2d748aaddae7"
server_key_version: "encryptionkey@0"
I20260924 16:02:51.043733 3400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "533bea1e6ba543a4a6e87e64dc2c39da"
format_stamp: "Formatted at 2026-09-24 16:02:51 on dist-test-slave-n106"
server_key: "72fc5ffc080cc821c47a7fd19398f4e3"
server_key_iv: "7480473f7ec14d116a4c2d748aaddae7"
server_key_version: "encryptionkey@0"
I20260924 16:02:51.047394 3400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 16:02:51.049979 3414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:51.051174 3400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:02:51.051312 3400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "533bea1e6ba543a4a6e87e64dc2c39da"
format_stamp: "Formatted at 2026-09-24 16:02:51 on dist-test-slave-n106"
server_key: "72fc5ffc080cc821c47a7fd19398f4e3"
server_key_iv: "7480473f7ec14d116a4c2d748aaddae7"
server_key_version: "encryptionkey@0"
I20260924 16:02:51.051417 3400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:51.055953 3400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:51.056596 3400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:51.056769 3400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:51.063510 3400 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:44077
I20260924 16:02:51.063511 3467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:44077 every 8 connection(s)
I20260924 16:02:51.064507 3400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:02:51.067710 3468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:51.069942 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 3400
I20260924 16:02:51.070092 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:02:51.070375 26036 external_mini_cluster.cc:1428] Setting key 58d675d62226e20bee5055fbb9b2dec9
I20260924 16:02:51.074448 3468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 533bea1e6ba543a4a6e87e64dc2c39da: Bootstrap starting.
I20260924 16:02:51.077080 3471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:51.071907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35368 (local address 127.25.109.62:44077)
0924 16:02:51.072655 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:51.072676 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:51.072723 (+ 47us) server_negotiation.cc:407] Connection header received
0924 16:02:51.073565 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:51.073590 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:51.074046 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:51.074579 (+ 533us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:51.075561 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:51.075567 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:51.075584 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:02:51.075600 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:51.075680 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:02:51.075706 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:51.075858 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:51.075926 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 16:02:51.076181 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":106,"threads_started":1}
I20260924 16:02:51.077226 3468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 533bea1e6ba543a4a6e87e64dc2c39da: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:51.077955 3468 log.cc:824] T 00000000000000000000000000000000 P 533bea1e6ba543a4a6e87e64dc2c39da: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:51.079859 3468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 533bea1e6ba543a4a6e87e64dc2c39da: No bootstrap required, opened a new log
I20260924 16:02:51.083254 3468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 533bea1e6ba543a4a6e87e64dc2c39da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "533bea1e6ba543a4a6e87e64dc2c39da" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44077 } }
I20260924 16:02:51.083478 3468 raft_consensus.cc:374] T 00000000000000000000000000000000 P 533bea1e6ba543a4a6e87e64dc2c39da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:51.083557 3468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 533bea1e6ba543a4a6e87e64dc2c39da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533bea1e6ba543a4a6e87e64dc2c39da, State: Initialized, Role: FOLLOWER
I20260924 16:02:51.083997 3468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 533bea1e6ba543a4a6e87e64dc2c39da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "533bea1e6ba543a4a6e87e64dc2c39da" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44077 } }
I20260924 16:02:51.084146 3468 raft_consensus.cc:388] T 00000000000000000000000000000000 P 533bea1e6ba543a4a6e87e64dc2c39da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:02:51.084209 3468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 533bea1e6ba543a4a6e87e64dc2c39da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:02:51.084326 3468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 533bea1e6ba543a4a6e87e64dc2c39da [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:51.085199 3468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 533bea1e6ba543a4a6e87e64dc2c39da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "533bea1e6ba543a4a6e87e64dc2c39da" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44077 } }
I20260924 16:02:51.085531 3468 leader_election.cc:302] T 00000000000000000000000000000000 P 533bea1e6ba543a4a6e87e64dc2c39da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 533bea1e6ba543a4a6e87e64dc2c39da; no voters:
I20260924 16:02:51.085827 3468 leader_election.cc:288] T 00000000000000000000000000000000 P 533bea1e6ba543a4a6e87e64dc2c39da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:02:51.085935 3473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 533bea1e6ba543a4a6e87e64dc2c39da [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:51.086194 3473 raft_consensus.cc:686] T 00000000000000000000000000000000 P 533bea1e6ba543a4a6e87e64dc2c39da [term 1 LEADER]: Becoming Leader. State: Replica: 533bea1e6ba543a4a6e87e64dc2c39da, State: Running, Role: LEADER
I20260924 16:02:51.086479 3473 consensus_queue.cc:237] T 00000000000000000000000000000000 P 533bea1e6ba543a4a6e87e64dc2c39da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "533bea1e6ba543a4a6e87e64dc2c39da" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44077 } }
I20260924 16:02:51.087023 3468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 533bea1e6ba543a4a6e87e64dc2c39da [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:02:51.088078 3474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 533bea1e6ba543a4a6e87e64dc2c39da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "533bea1e6ba543a4a6e87e64dc2c39da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "533bea1e6ba543a4a6e87e64dc2c39da" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44077 } } }
I20260924 16:02:51.088219 3474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 533bea1e6ba543a4a6e87e64dc2c39da [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:51.088526 3475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 533bea1e6ba543a4a6e87e64dc2c39da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 533bea1e6ba543a4a6e87e64dc2c39da. Latest consensus state: current_term: 1 leader_uuid: "533bea1e6ba543a4a6e87e64dc2c39da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "533bea1e6ba543a4a6e87e64dc2c39da" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44077 } } }
I20260924 16:02:51.088619 3475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 533bea1e6ba543a4a6e87e64dc2c39da [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:51.088713 3482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:02:51.091356 3482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:02:51.097189 3482 catalog_manager.cc:1347] Generated new cluster ID: f7137eb0368f4fd9882bed50cf40ea57
I20260924 16:02:51.097252 3482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:02:51.108860 3482 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:02:51.109177 3482 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:02:51.117079 3482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 533bea1e6ba543a4a6e87e64dc2c39da: Generated new TSK 0
I20260924 16:02:51.117825 3482 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:02:51.122174 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:0
--local_ip_for_outbound_sockets=127.25.109.1
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:44077
--builtin_ntp_servers=127.25.109.20:45039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 16:02:51.234570 3492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:51.234822 3492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:51.234861 3492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:51.238951 3492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:51.239010 3492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:51.239068 3492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:02:51.243880 3492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:45039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:44077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:51.244983 3492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:51.246362 3492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:51.256556 3497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:51.256572 3498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:51.256624 3500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:51.256995 3492 server_base.cc:1034] running on GCE node
I20260924 16:02:51.257397 3492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:51.258023 3492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:51.259184 3492 hybrid_clock.cc:648] HybridClock initialized: now 1790265771259150 us; error 47 us; skew 500 ppm
I20260924 16:02:51.260991 3492 webserver.cc:460] Webserver started at http://127.25.109.1:34895/ using document root <none> and password file <none>
I20260924 16:02:51.261619 3492 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:51.261683 3492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:51.261900 3492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:51.263989 3492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/data/instance:
uuid: "b83b9434f9f04f8d930458381020ab23"
format_stamp: "Formatted at 2026-09-24 16:02:51 on dist-test-slave-n106"
server_key: "d30ce17dc5d05210bb214aef201e4896"
server_key_iv: "12a60ed691398ef3861e2ee3a092f13b"
server_key_version: "encryptionkey@0"
I20260924 16:02:51.264645 3492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance:
uuid: "b83b9434f9f04f8d930458381020ab23"
format_stamp: "Formatted at 2026-09-24 16:02:51 on dist-test-slave-n106"
server_key: "d30ce17dc5d05210bb214aef201e4896"
server_key_iv: "12a60ed691398ef3861e2ee3a092f13b"
server_key_version: "encryptionkey@0"
I20260924 16:02:51.268388 3492 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:02:51.270923 3506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:51.272081 3492 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 16:02:51.272212 3492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "b83b9434f9f04f8d930458381020ab23"
format_stamp: "Formatted at 2026-09-24 16:02:51 on dist-test-slave-n106"
server_key: "d30ce17dc5d05210bb214aef201e4896"
server_key_iv: "12a60ed691398ef3861e2ee3a092f13b"
server_key_version: "encryptionkey@0"
I20260924 16:02:51.272316 3492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:51.276690 3492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:51.277264 3492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:51.277446 3492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:51.277954 3492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:51.278834 3492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:51.278903 3492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:51.278990 3492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:51.279039 3492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:51.287297 3492 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:40501
I20260924 16:02:51.287315 3620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:40501 every 8 connection(s)
I20260924 16:02:51.288161 3492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:02:51.294072 3471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:51.290072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:39917 (local address 127.25.109.62:44077)
0924 16:02:51.290231 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:51.290235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:51.291249 (+ 1014us) server_negotiation.cc:407] Connection header received
0924 16:02:51.292546 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:51.292549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:51.292586 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:51.292662 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:51.293726 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:51.293728 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:51.293730 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:02:51.293738 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:51.293753 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:02:51.293760 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:51.293827 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:51.293935 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 16:02:51.293971 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 16:02:51.294866 3623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:51.290476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:44077 (local address 127.25.109.1:39917)
0924 16:02:51.291090 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:51.291147 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:51.292297 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:51.292790 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:51.292796 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:51.292878 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:51.293487 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:02:51.293489 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:51.293529 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:02:51.293598 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:51.293828 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:51.293832 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:51.293836 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 16:02:51.293954 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 16:02:51.294143 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":100,"threads_started":1}
I20260924 16:02:51.296036 3621 heartbeater.cc:347] Connected to a master server at 127.25.109.62:44077
I20260924 16:02:51.296267 3621 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:51.296706 3621 heartbeater.cc:511] Master 127.25.109.62:44077 requested a full tablet report, sending...
I20260924 16:02:51.298174 3432 ts_manager.cc:194] Registered new tserver with Master: b83b9434f9f04f8d930458381020ab23 (127.25.109.1:40501)
I20260924 16:02:51.298444 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 3492
I20260924 16:02:51.298575 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance
I20260924 16:02:51.298871 26036 external_mini_cluster.cc:1428] Setting key f926cb57effa783a910b60c50a3462bc
I20260924 16:02:51.300815 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:0
--local_ip_for_outbound_sockets=127.25.109.2
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:44077
--builtin_ntp_servers=127.25.109.20:45039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 16:02:51.409821 3624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:51.410130 3624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:51.410212 3624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:51.414225 3624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:51.414306 3624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:51.414394 3624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:02:51.422272 3624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:45039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:44077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:51.423727 3624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:51.425376 3624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:02:51.435350 3624 server_base.cc:1034] running on GCE node
W20260924 16:02:51.435292 3629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:51.435286 3630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:51.435302 3632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:51.436008 3624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:51.436749 3624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:51.437947 3624 hybrid_clock.cc:648] HybridClock initialized: now 1790265771437926 us; error 31 us; skew 500 ppm
I20260924 16:02:51.439867 3624 webserver.cc:460] Webserver started at http://127.25.109.2:43091/ using document root <none> and password file <none>
I20260924 16:02:51.440529 3624 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:51.440598 3624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:51.440856 3624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:51.442894 3624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/data/instance:
uuid: "25de59bea6384266a0c92030553d5301"
format_stamp: "Formatted at 2026-09-24 16:02:51 on dist-test-slave-n106"
server_key: "5a1a4f2c1de96b4f60dff50879340421"
server_key_iv: "18eb48120801424e845e3f7c5f9868b6"
server_key_version: "encryptionkey@0"
I20260924 16:02:51.443483 3624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance:
uuid: "25de59bea6384266a0c92030553d5301"
format_stamp: "Formatted at 2026-09-24 16:02:51 on dist-test-slave-n106"
server_key: "5a1a4f2c1de96b4f60dff50879340421"
server_key_iv: "18eb48120801424e845e3f7c5f9868b6"
server_key_version: "encryptionkey@0"
I20260924 16:02:51.447142 3624 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 16:02:51.449683 3638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:51.450852 3624 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:02:51.450991 3624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "25de59bea6384266a0c92030553d5301"
format_stamp: "Formatted at 2026-09-24 16:02:51 on dist-test-slave-n106"
server_key: "5a1a4f2c1de96b4f60dff50879340421"
server_key_iv: "18eb48120801424e845e3f7c5f9868b6"
server_key_version: "encryptionkey@0"
I20260924 16:02:51.451119 3624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:51.455675 3624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:51.456305 3624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:51.456509 3624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:51.457067 3624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:51.457993 3624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:51.458072 3624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:51.458145 3624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:51.458196 3624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:51.467005 3624 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:43393
I20260924 16:02:51.467033 3753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:43393 every 8 connection(s)
I20260924 16:02:51.467950 3624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:02:51.474349 3471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:51.469783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:39535 (local address 127.25.109.62:44077)
0924 16:02:51.469936 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:51.469940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:51.471124 (+ 1184us) server_negotiation.cc:407] Connection header received
0924 16:02:51.472567 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:51.472570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:51.472618 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:51.472704 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:51.473850 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:51.473854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:51.473857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:02:51.473870 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:51.473894 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:02:51.473909 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:51.474008 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:51.474154 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 16:02:51.474204 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 16:02:51.475304 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:51.470211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:44077 (local address 127.25.109.2:39535)
0924 16:02:51.470935 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:51.470978 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:51.472257 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:51.472872 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:51.472879 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:51.472969 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:51.473571 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:02:51.473573 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:51.473621 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:02:51.473692 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:51.473991 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:51.474000 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:51.474006 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 16:02:51.474200 (+ 194us) client_negotiation.cc:240] Negotiation successful
0924 16:02:51.474449 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":115,"threads_started":1}
I20260924 16:02:51.476539 3754 heartbeater.cc:347] Connected to a master server at 127.25.109.62:44077
I20260924 16:02:51.476787 3754 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:51.476986 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 3624
I20260924 16:02:51.477119 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance
I20260924 16:02:51.477294 3754 heartbeater.cc:511] Master 127.25.109.62:44077 requested a full tablet report, sending...
I20260924 16:02:51.477454 26036 external_mini_cluster.cc:1428] Setting key 7030650637c341654af5df22531e2e0b
I20260924 16:02:51.478411 3430 ts_manager.cc:194] Registered new tserver with Master: 25de59bea6384266a0c92030553d5301 (127.25.109.2:43393)
I20260924 16:02:51.479310 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:0
--local_ip_for_outbound_sockets=127.25.109.3
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:44077
--builtin_ntp_servers=127.25.109.20:45039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 16:02:51.591717 3757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:51.591951 3757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:51.592020 3757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:51.595983 3757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:51.596035 3757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:51.596123 3757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:02:51.600795 3757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:45039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:44077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:51.601855 3757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:51.603212 3757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:51.612721 3765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:51.612728 3763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:51.612773 3762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:51.612939 3757 server_base.cc:1034] running on GCE node
I20260924 16:02:51.613514 3757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:51.614166 3757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:51.615357 3757 hybrid_clock.cc:648] HybridClock initialized: now 1790265771615331 us; error 38 us; skew 500 ppm
I20260924 16:02:51.617606 3757 webserver.cc:460] Webserver started at http://127.25.109.3:35395/ using document root <none> and password file <none>
I20260924 16:02:51.618196 3757 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:51.618285 3757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:51.618520 3757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:51.620704 3757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/data/instance:
uuid: "c437c0cdebac4c79be4d8c342f3c5ee5"
format_stamp: "Formatted at 2026-09-24 16:02:51 on dist-test-slave-n106"
server_key: "d94d14184b83d957b2d53e90df6c22e4"
server_key_iv: "1709847ad7e987b0cfcd2fcf59223029"
server_key_version: "encryptionkey@0"
I20260924 16:02:51.621280 3757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance:
uuid: "c437c0cdebac4c79be4d8c342f3c5ee5"
format_stamp: "Formatted at 2026-09-24 16:02:51 on dist-test-slave-n106"
server_key: "d94d14184b83d957b2d53e90df6c22e4"
server_key_iv: "1709847ad7e987b0cfcd2fcf59223029"
server_key_version: "encryptionkey@0"
I20260924 16:02:51.624965 3757 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:02:51.627540 3771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:51.628794 3757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:02:51.628924 3757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "c437c0cdebac4c79be4d8c342f3c5ee5"
format_stamp: "Formatted at 2026-09-24 16:02:51 on dist-test-slave-n106"
server_key: "d94d14184b83d957b2d53e90df6c22e4"
server_key_iv: "1709847ad7e987b0cfcd2fcf59223029"
server_key_version: "encryptionkey@0"
I20260924 16:02:51.629053 3757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:51.633544 3757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:51.634135 3757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:51.634305 3757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:51.634881 3757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:51.635761 3757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:51.635830 3757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:51.635897 3757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:51.635944 3757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:51.644244 3757 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:46475
I20260924 16:02:51.644264 3884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:46475 every 8 connection(s)
I20260924 16:02:51.645190 3757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:02:51.651180 3471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:51.646970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:49859 (local address 127.25.109.62:44077)
0924 16:02:51.647114 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:51.647117 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:51.648079 (+ 962us) server_negotiation.cc:407] Connection header received
0924 16:02:51.649568 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:51.649571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:51.649611 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:51.649709 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:51.650783 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:51.650786 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:51.650788 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:02:51.650801 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:51.650822 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:02:51.650830 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:51.650915 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:51.651033 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 16:02:51.651072 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:02:51.651978 3887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:51.647337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:44077 (local address 127.25.109.3:49859)
0924 16:02:51.647960 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:51.647997 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:51.649218 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:51.649867 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:51.649874 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:51.649959 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:51.650505 (+ 546us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:02:51.650507 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:51.650546 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:02:51.650613 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:51.650911 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:51.650917 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:51.650921 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 16:02:51.651045 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 16:02:51.651225 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":103,"threads_started":1}
I20260924 16:02:51.653111 3885 heartbeater.cc:347] Connected to a master server at 127.25.109.62:44077
I20260924 16:02:51.653343 3885 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:51.653738 3885 heartbeater.cc:511] Master 127.25.109.62:44077 requested a full tablet report, sending...
I20260924 16:02:51.654832 3430 ts_manager.cc:194] Registered new tserver with Master: c437c0cdebac4c79be4d8c342f3c5ee5 (127.25.109.3:46475)
I20260924 16:02:51.655184 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 3757
I20260924 16:02:51.655320 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance
I20260924 16:02:51.655699 26036 external_mini_cluster.cc:1428] Setting key f3673e3261a9f37d98ff14baf54608ce
I20260924 16:02:51.657696 26036 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 16:02:51.660364 3471 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:02:51.659589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35378 (local address 127.25.109.62:44077)
0924 16:02:51.659722 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:51.659726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:51.659910 (+ 184us) server_negotiation.cc:407] Connection header received
0924 16:02:51.659989 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:51.659991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:51.660017 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:51.660077 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:51.660276 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35378: BlockingRecv error: recv got EOF from 127.0.0.1:35378 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 16:02:51.660702 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 3492
I20260924 16:02:51.669090 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 3624
I20260924 16:02:51.675009 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 3757
I20260924 16:02:51.680374 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 3400
2026-09-24T16:02:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (804 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:02:51 dist-test-slave-n106 krb5kdc[3895](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:02:51 dist-test-slave-n106 krb5kdc[3895](info): set up 2 sockets
Sep 24 16:02:51 dist-test-slave-n106 krb5kdc[3895](info): commencing operation
krb5kdc: starting...
W20260924 16:02:52.722462 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:02:52 dist-test-slave-n106 krb5kdc[3895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:02:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:02:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
I20260924 16:02:52.845108 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:33653
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:38547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:33653
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:52.955400 3908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:52.955749 3908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:52.955832 3908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:52.959875 3908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:02:52.959978 3908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:52.960013 3908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:52.960055 3908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:02:52.960086 3908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:02:52.965520 3908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:38547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:33653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:33653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:52.966758 3908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:52.967808 3908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:52.976742 3913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:52.976704 3916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:52.976926 3908 server_base.cc:1034] running on GCE node
W20260924 16:02:52.976713 3914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:52.977584 3908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:52.978632 3908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:52.979832 3908 hybrid_clock.cc:648] HybridClock initialized: now 1790265772979809 us; error 42 us; skew 500 ppm
Sep 24 16:02:52 dist-test-slave-n106 krb5kdc[3895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265772, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:52.988176 3908 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:02:52.989156 3908 webserver.cc:460] Webserver started at http://127.25.109.62:35111/ using document root <none> and password file <none>
I20260924 16:02:52.989709 3908 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:52.989769 3908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:52.989935 3908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:52.991997 3908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "955c1bd155954d4892b6f778b02b44e2"
format_stamp: "Formatted at 2026-09-24 16:02:52 on dist-test-slave-n106"
server_key: "cc11f334288b2ddbfc5ae396891396df"
server_key_iv: "65607c97098ab249b5f36fa7d594e0f3"
server_key_version: "encryptionkey@0"
I20260924 16:02:52.992588 3908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "955c1bd155954d4892b6f778b02b44e2"
format_stamp: "Formatted at 2026-09-24 16:02:52 on dist-test-slave-n106"
server_key: "cc11f334288b2ddbfc5ae396891396df"
server_key_iv: "65607c97098ab249b5f36fa7d594e0f3"
server_key_version: "encryptionkey@0"
I20260924 16:02:52.996284 3908 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:02:52.998744 3923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:52.999969 3908 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:02:53.000106 3908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "955c1bd155954d4892b6f778b02b44e2"
format_stamp: "Formatted at 2026-09-24 16:02:52 on dist-test-slave-n106"
server_key: "cc11f334288b2ddbfc5ae396891396df"
server_key_iv: "65607c97098ab249b5f36fa7d594e0f3"
server_key_version: "encryptionkey@0"
I20260924 16:02:53.000213 3908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:53.020552 3908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:53.023343 3908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:53.023572 3908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:53.030484 3908 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:33653
I20260924 16:02:53.030511 3975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:33653 every 8 connection(s)
I20260924 16:02:53.031435 3908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:02:53.034253 3976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:53.039675 3976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 955c1bd155954d4892b6f778b02b44e2: Bootstrap starting.
I20260924 16:02:53.041329 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 3908
I20260924 16:02:53.041451 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:02:53.041759 26036 external_mini_cluster.cc:1428] Setting key e63bd91e02a107f1d670c9bca339bcf5
I20260924 16:02:53.042500 3976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 955c1bd155954d4892b6f778b02b44e2: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:53.043359 3976 log.cc:824] T 00000000000000000000000000000000 P 955c1bd155954d4892b6f778b02b44e2: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:53.045583 3976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 955c1bd155954d4892b6f778b02b44e2: No bootstrap required, opened a new log
I20260924 16:02:53.049912 3976 raft_consensus.cc:348] T 00000000000000000000000000000000 P 955c1bd155954d4892b6f778b02b44e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955c1bd155954d4892b6f778b02b44e2" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33653 } }
I20260924 16:02:53.050212 3976 raft_consensus.cc:374] T 00000000000000000000000000000000 P 955c1bd155954d4892b6f778b02b44e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:53.050302 3976 raft_consensus.cc:729] T 00000000000000000000000000000000 P 955c1bd155954d4892b6f778b02b44e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 955c1bd155954d4892b6f778b02b44e2, State: Initialized, Role: FOLLOWER
I20260924 16:02:53.050727 3976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 955c1bd155954d4892b6f778b02b44e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955c1bd155954d4892b6f778b02b44e2" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33653 } }
Sep 24 16:02:53 dist-test-slave-n106 krb5kdc[3895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:02:53.050845 3976 raft_consensus.cc:388] T 00000000000000000000000000000000 P 955c1bd155954d4892b6f778b02b44e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:02:53.050930 3976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 955c1bd155954d4892b6f778b02b44e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:02:53.051023 3976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 955c1bd155954d4892b6f778b02b44e2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:53.051877 3976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 955c1bd155954d4892b6f778b02b44e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955c1bd155954d4892b6f778b02b44e2" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33653 } }
I20260924 16:02:53.052258 3976 leader_election.cc:302] T 00000000000000000000000000000000 P 955c1bd155954d4892b6f778b02b44e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 955c1bd155954d4892b6f778b02b44e2; no voters:
I20260924 16:02:53.052593 3976 leader_election.cc:288] T 00000000000000000000000000000000 P 955c1bd155954d4892b6f778b02b44e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:02:53.052768 3981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 955c1bd155954d4892b6f778b02b44e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:53.052990 3981 raft_consensus.cc:686] T 00000000000000000000000000000000 P 955c1bd155954d4892b6f778b02b44e2 [term 1 LEADER]: Becoming Leader. State: Replica: 955c1bd155954d4892b6f778b02b44e2, State: Running, Role: LEADER
I20260924 16:02:53.053299 3981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 955c1bd155954d4892b6f778b02b44e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955c1bd155954d4892b6f778b02b44e2" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33653 } }
I20260924 16:02:53.053929 3976 sys_catalog.cc:564] T 00000000000000000000000000000000 P 955c1bd155954d4892b6f778b02b44e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:02:53.054733 3983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 955c1bd155954d4892b6f778b02b44e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 955c1bd155954d4892b6f778b02b44e2. Latest consensus state: current_term: 1 leader_uuid: "955c1bd155954d4892b6f778b02b44e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955c1bd155954d4892b6f778b02b44e2" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33653 } } }
I20260924 16:02:53.054926 3983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 955c1bd155954d4892b6f778b02b44e2 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:53.055213 3982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 955c1bd155954d4892b6f778b02b44e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "955c1bd155954d4892b6f778b02b44e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955c1bd155954d4892b6f778b02b44e2" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33653 } } }
I20260924 16:02:53.055311 3982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 955c1bd155954d4892b6f778b02b44e2 [sys.catalog]: This master's current role is: LEADER
I20260924 16:02:53.056609 3990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:02:53.057116 3979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:53.042883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47476 (local address 127.25.109.62:33653)
0924 16:02:53.043305 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:53.043325 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:53.043359 (+ 34us) server_negotiation.cc:407] Connection header received
0924 16:02:53.044243 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:53.044263 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:53.044650 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:53.044923 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:53.045812 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.046762 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.047922 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.048382 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.048594 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:53.051481 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:53.051486 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:53.051501 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:53.051530 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:53.053656 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:53.054139 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:53.054145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:53.054150 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:53.054289 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:53.054631 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:53.054636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:53.054637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:53.055173 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:53.055347 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:53.055656 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 16:02:53.055941 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":147,"threads_started":1}
I20260924 16:02:53.059751 3990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:02:53.065258 3990 catalog_manager.cc:1347] Generated new cluster ID: e2a47d579a644f86bdf293b4bdf2916c
I20260924 16:02:53.065318 3990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:02:53.083454 3990 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:02:53.084793 3990 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:02:53.095472 3990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 955c1bd155954d4892b6f778b02b44e2: Generated new TSK 0
I20260924 16:02:53.096272 3990 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
I20260924 16:02:53.127274 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:0
--local_ip_for_outbound_sockets=127.25.109.1
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:33653
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:38547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:53.241288 4002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:53.241580 4002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:53.241624 4002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:53.245633 4002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:53.245699 4002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:53.245759 4002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:02:53.251155 4002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:38547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:33653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:53.252255 4002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:53.253382 4002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:53.263401 4008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:53.263405 4007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:53.263401 4010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:53.263698 4002 server_base.cc:1034] running on GCE node
I20260924 16:02:53.264195 4002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:53.264904 4002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:53.266094 4002 hybrid_clock.cc:648] HybridClock initialized: now 1790265773266070 us; error 56 us; skew 500 ppm
Sep 24 16:02:53 dist-test-slave-n106 krb5kdc[3895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265773, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:53.269186 4002 init.cc:377] Logged in from keytab as kudu/127.25.109.1@KRBTEST.COM (short username kudu)
I20260924 16:02:53.270143 4002 webserver.cc:460] Webserver started at http://127.25.109.1:42699/ using document root <none> and password file <none>
I20260924 16:02:53.270768 4002 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:53.270866 4002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:53.271126 4002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:53.273392 4002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/data/instance:
uuid: "d77bfffcb5ae49eda017aa7874898da9"
format_stamp: "Formatted at 2026-09-24 16:02:53 on dist-test-slave-n106"
server_key: "33688d1d05f2606c5d0124ea79181dd1"
server_key_iv: "6c195caa12bd4dd8a2521dc4e8fcd14b"
server_key_version: "encryptionkey@0"
I20260924 16:02:53.273988 4002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance:
uuid: "d77bfffcb5ae49eda017aa7874898da9"
format_stamp: "Formatted at 2026-09-24 16:02:53 on dist-test-slave-n106"
server_key: "33688d1d05f2606c5d0124ea79181dd1"
server_key_iv: "6c195caa12bd4dd8a2521dc4e8fcd14b"
server_key_version: "encryptionkey@0"
I20260924 16:02:53.277897 4002 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:02:53.280368 4017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:53.281603 4002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 16:02:53.281755 4002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "d77bfffcb5ae49eda017aa7874898da9"
format_stamp: "Formatted at 2026-09-24 16:02:53 on dist-test-slave-n106"
server_key: "33688d1d05f2606c5d0124ea79181dd1"
server_key_iv: "6c195caa12bd4dd8a2521dc4e8fcd14b"
server_key_version: "encryptionkey@0"
I20260924 16:02:53.281873 4002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:53.315403 4002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:53.318429 4002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:53.318657 4002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:53.319271 4002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:53.320120 4002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:53.320201 4002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:53.320295 4002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:53.320346 4002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:53.329274 4002 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:37911
I20260924 16:02:53.329301 4130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:37911 every 8 connection(s)
I20260924 16:02:53.330164 4002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:02:53.333964 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 4002
I20260924 16:02:53.334097 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance
I20260924 16:02:53.334401 26036 external_mini_cluster.cc:1428] Setting key 1942a7372fd84a46772b0ec0533237fb
Sep 24 16:02:53 dist-test-slave-n106 krb5kdc[3895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265773, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.109.2@KRBTEST.COM; defaulting to no policy
I20260924 16:02:53.346302 3979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:53.331944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:55623 (local address 127.25.109.62:33653)
0924 16:02:53.332113 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:53.332117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:53.332978 (+ 861us) server_negotiation.cc:407] Connection header received
0924 16:02:53.334726 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:53.334730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:53.334802 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:53.334919 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:53.336220 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.336880 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.338860 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.339089 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.339176 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:53.342494 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:53.342497 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:53.342499 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:53.342523 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:53.344017 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:53.344740 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:53.344743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:53.344745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:53.344837 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:53.345249 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:53.345252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:53.345253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:53.345423 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:53.345518 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:53.346059 (+ 541us) server_negotiation.cc:299] Negotiation successful
0924 16:02:53.346171 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.2@KRBTEST.COM" created.
I20260924 16:02:53.347409 4134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:53.332312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33653 (local address 127.25.109.1:55623)
0924 16:02:53.332818 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:53.332863 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:53.334490 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:53.335119 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:53.335127 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:53.335518 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:53.336065 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:53.336080 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.337042 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.337046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:53.338722 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:53.338738 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.339258 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.339266 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:53.339538 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:53.340261 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:53.340277 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:53.342311 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:53.344148 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:53.344157 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:53.344175 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:53.344623 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:53.344947 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:53.344954 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:53.344958 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:53.345144 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:53.345523 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:53.345529 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:53.345821 (+ 292us) client_negotiation.cc:769] Sending connection context
0924 16:02:53.345993 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 16:02:53.346264 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":120,"threads_started":1}
I20260924 16:02:53.348893 4132 heartbeater.cc:347] Connected to a master server at 127.25.109.62:33653
I20260924 16:02:53.349220 4132 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:53.349898 4132 heartbeater.cc:511] Master 127.25.109.62:33653 requested a full tablet report, sending...
I20260924 16:02:53.351502 3940 ts_manager.cc:194] Registered new tserver with Master: d77bfffcb5ae49eda017aa7874898da9 (127.25.109.1:37911)
I20260924 16:02:53.352751 3940 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.1@KRBTEST.COM'} at 127.25.109.1:55623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
I20260924 16:02:53.363256 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:0
--local_ip_for_outbound_sockets=127.25.109.2
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:33653
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:38547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:53.474993 4137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:53.475306 4137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:53.475399 4137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:53.479482 4137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:53.479570 4137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:53.479662 4137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:02:53.485035 4137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:38547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:33653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:53.486209 4137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:53.487252 4137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:53.496927 4142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:53.496923 4145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:53.496920 4143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:53.496994 4137 server_base.cc:1034] running on GCE node
I20260924 16:02:53.497602 4137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:53.498286 4137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:53.499506 4137 hybrid_clock.cc:648] HybridClock initialized: now 1790265773499470 us; error 46 us; skew 500 ppm
Sep 24 16:02:53 dist-test-slave-n106 krb5kdc[3895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265773, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:53.502368 4137 init.cc:377] Logged in from keytab as kudu/127.25.109.2@KRBTEST.COM (short username kudu)
I20260924 16:02:53.503341 4137 webserver.cc:460] Webserver started at http://127.25.109.2:45501/ using document root <none> and password file <none>
I20260924 16:02:53.503922 4137 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:53.504024 4137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:53.504237 4137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:53.506357 4137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/data/instance:
uuid: "c11e510346784980a5ff729253e0de35"
format_stamp: "Formatted at 2026-09-24 16:02:53 on dist-test-slave-n106"
server_key: "a8cb32f80aa6d5277c694856dd9ac6bf"
server_key_iv: "9d4ceb3d00027b6f713201123a989c0d"
server_key_version: "encryptionkey@0"
I20260924 16:02:53.506884 4137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance:
uuid: "c11e510346784980a5ff729253e0de35"
format_stamp: "Formatted at 2026-09-24 16:02:53 on dist-test-slave-n106"
server_key: "a8cb32f80aa6d5277c694856dd9ac6bf"
server_key_iv: "9d4ceb3d00027b6f713201123a989c0d"
server_key_version: "encryptionkey@0"
I20260924 16:02:53.510571 4137 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 16:02:53.512961 4152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:53.514210 4137 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:02:53.514353 4137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "c11e510346784980a5ff729253e0de35"
format_stamp: "Formatted at 2026-09-24 16:02:53 on dist-test-slave-n106"
server_key: "a8cb32f80aa6d5277c694856dd9ac6bf"
server_key_iv: "9d4ceb3d00027b6f713201123a989c0d"
server_key_version: "encryptionkey@0"
I20260924 16:02:53.514464 4137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:53.530530 4137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:53.533735 4137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:53.533948 4137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:53.534588 4137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:53.535471 4137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:53.535529 4137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:53.535600 4137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:53.535634 4137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:53.544725 4137 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:46649
I20260924 16:02:53.544745 4266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:46649 every 8 connection(s)
I20260924 16:02:53.545693 4137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:02:53.549817 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 4137
I20260924 16:02:53.549959 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance
I20260924 16:02:53.550256 26036 external_mini_cluster.cc:1428] Setting key 82e118d2208cff0d5643627cf7b0ec95
Sep 24 16:02:53 dist-test-slave-n106 krb5kdc[3895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265773, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.109.3@KRBTEST.COM; defaulting to no policy
I20260924 16:02:53.560659 3979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:53.547505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:49961 (local address 127.25.109.62:33653)
0924 16:02:53.547680 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:53.547685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:53.548594 (+ 909us) server_negotiation.cc:407] Connection header received
0924 16:02:53.549994 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:53.549998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:53.550061 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:53.550167 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:53.551555 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.552081 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.553449 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.553749 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.553839 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:53.556987 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:53.556991 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:53.556993 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:53.557019 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:53.558450 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:53.558994 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:53.558998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:53.559000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:53.559084 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:53.559433 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:53.559437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:53.559438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:53.559625 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:53.559729 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:53.560313 (+ 584us) server_negotiation.cc:299] Negotiation successful
0924 16:02:53.560475 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 16:02:53.561499 4269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:53.547929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33653 (local address 127.25.109.2:49961)
0924 16:02:53.548382 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:53.548481 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:53.549762 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:53.550379 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:53.550389 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:53.550893 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:53.551398 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:53.551414 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.552214 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.552218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:53.553310 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:53.553326 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.553914 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.553919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:53.554159 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:53.554865 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:53.554891 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:53.556808 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:53.558585 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:53.558591 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:53.558607 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:53.558883 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:53.559185 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:53.559188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:53.559190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:53.559333 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:53.559729 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:53.559737 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:53.560045 (+ 308us) client_negotiation.cc:769] Sending connection context
0924 16:02:53.560249 (+ 204us) client_negotiation.cc:240] Negotiation successful
0924 16:02:53.560603 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.3@KRBTEST.COM" created.
I20260924 16:02:53.562897 4267 heartbeater.cc:347] Connected to a master server at 127.25.109.62:33653
I20260924 16:02:53.563189 4267 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:53.563795 4267 heartbeater.cc:511] Master 127.25.109.62:33653 requested a full tablet report, sending...
I20260924 16:02:53.565057 3940 ts_manager.cc:194] Registered new tserver with Master: c11e510346784980a5ff729253e0de35 (127.25.109.2:46649)
I20260924 16:02:53.565646 3940 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.2@KRBTEST.COM'} at 127.25.109.2:49961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
I20260924 16:02:53.577818 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:0
--local_ip_for_outbound_sockets=127.25.109.3
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:33653
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:38547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:02:53.689576 4272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:02:53.689821 4272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:02:53.689863 4272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:02:53.693699 4272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:02:53.693746 4272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:02:53.693801 4272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:02:53.698869 4272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:38547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:33653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:02:53.699820 4272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:02:53.700785 4272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:02:53.710083 4277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:53.710107 4278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:02:53.710083 4280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:02:53.710264 4272 server_base.cc:1034] running on GCE node
I20260924 16:02:53.710801 4272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:02:53.711403 4272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:02:53.712548 4272 hybrid_clock.cc:648] HybridClock initialized: now 1790265773712529 us; error 29 us; skew 500 ppm
Sep 24 16:02:53 dist-test-slave-n106 krb5kdc[3895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265773, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:02:53.715067 4272 init.cc:377] Logged in from keytab as kudu/127.25.109.3@KRBTEST.COM (short username kudu)
I20260924 16:02:53.715833 4272 webserver.cc:460] Webserver started at http://127.25.109.3:44495/ using document root <none> and password file <none>
I20260924 16:02:53.716339 4272 fs_manager.cc:359] Metadata directory not provided
I20260924 16:02:53.716394 4272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:02:53.716639 4272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:02:53.718657 4272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/data/instance:
uuid: "dff99776a0464b77b19bce7fdcc467b6"
format_stamp: "Formatted at 2026-09-24 16:02:53 on dist-test-slave-n106"
server_key: "5eea0b2141704cb2e71bb01eb7a28a87"
server_key_iv: "6341f1235242633835289fa1c7b8b6ea"
server_key_version: "encryptionkey@0"
I20260924 16:02:53.719161 4272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance:
uuid: "dff99776a0464b77b19bce7fdcc467b6"
format_stamp: "Formatted at 2026-09-24 16:02:53 on dist-test-slave-n106"
server_key: "5eea0b2141704cb2e71bb01eb7a28a87"
server_key_iv: "6341f1235242633835289fa1c7b8b6ea"
server_key_version: "encryptionkey@0"
I20260924 16:02:53.722769 4272 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260924 16:02:53.725324 4287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:53.726498 4272 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260924 16:02:53.726629 4272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "dff99776a0464b77b19bce7fdcc467b6"
format_stamp: "Formatted at 2026-09-24 16:02:53 on dist-test-slave-n106"
server_key: "5eea0b2141704cb2e71bb01eb7a28a87"
server_key_iv: "6341f1235242633835289fa1c7b8b6ea"
server_key_version: "encryptionkey@0"
I20260924 16:02:53.726734 4272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:02:53.745883 4272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:02:53.748772 4272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:02:53.748970 4272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:02:53.749588 4272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:02:53.750411 4272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:02:53.750455 4272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:53.750519 4272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:02:53.750552 4272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:02:53.759709 4272 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:35991
I20260924 16:02:53.759773 4400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:35991 every 8 connection(s)
I20260924 16:02:53.760670 4272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:02:53.763921 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 4272
I20260924 16:02:53.764020 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance
I20260924 16:02:53.764320 26036 external_mini_cluster.cc:1428] Setting key 74c0210b6b5a6698cd319a349d88a0ad
Sep 24 16:02:53 dist-test-slave-n106 krb5kdc[3895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265773, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:02:53.774225 3979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:53.762465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:58395 (local address 127.25.109.62:33653)
0924 16:02:53.762622 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:53.762626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:53.763360 (+ 734us) server_negotiation.cc:407] Connection header received
0924 16:02:53.764891 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:53.764895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:53.764951 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:53.765039 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:53.766498 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.767126 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.768327 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.768607 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.768691 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:53.771054 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:53.771057 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:53.771059 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:53.771081 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:53.772302 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:53.772824 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:53.772827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:53.772828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:53.772900 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:53.773209 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:53.773212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:53.773213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:53.773379 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:53.773451 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:53.774001 (+ 550us) server_negotiation.cc:299] Negotiation successful
0924 16:02:53.774093 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 16:02:53.775035 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:53.762771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33653 (local address 127.25.109.3:58395)
0924 16:02:53.763204 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:53.763247 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:53.764708 (+ 1461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:53.765345 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:53.765352 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:53.765707 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:02:53.766346 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:53.766366 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.767239 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.767242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:53.768220 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:53.768230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.768734 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.768737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:53.768878 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:53.769509 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:02:53.769526 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:02:53.770898 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:02:53.772443 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:53.772453 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:53.772477 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:53.772738 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:53.772981 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:02:53.772983 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:02:53.772985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:02:53.773127 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:02:53.773463 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:02:53.773467 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:02:53.773759 (+ 292us) client_negotiation.cc:769] Sending connection context
0924 16:02:53.773956 (+ 197us) client_negotiation.cc:240] Negotiation successful
0924 16:02:53.774218 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":113,"threads_started":1}
I20260924 16:02:53.776182 4401 heartbeater.cc:347] Connected to a master server at 127.25.109.62:33653
I20260924 16:02:53.776469 4401 heartbeater.cc:464] Registering TS with master...
I20260924 16:02:53.776947 4401 heartbeater.cc:511] Master 127.25.109.62:33653 requested a full tablet report, sending...
I20260924 16:02:53.778132 3940 ts_manager.cc:194] Registered new tserver with Master: dff99776a0464b77b19bce7fdcc467b6 (127.25.109.3:35991)
I20260924 16:02:53.778745 3940 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.3@KRBTEST.COM'} at 127.25.109.3:58395
I20260924 16:02:53.789938 26036 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:02:53.799062 3979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:53.791822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47478 (local address 127.25.109.62:33653)
0924 16:02:53.791948 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:53.791951 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:53.792150 (+ 199us) server_negotiation.cc:407] Connection header received
0924 16:02:53.792259 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:53.792261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:53.792300 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:53.792366 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:02:53.793102 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.793597 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.794435 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.794659 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.794732 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:53.796087 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:02:53.796089 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:02:53.796091 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:02:53.796107 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:02:53.797577 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:53.797948 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:53.797951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:53.797952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:53.798031 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:02:53.798252 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:02:53.798255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:02:53.798256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:02:53.798453 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:02:53.798551 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:53.798748 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 16:02:53.798834 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 16:02:53.801839 3940 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:02:53.805037 3940 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:02:53.819485 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:53.813660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:46649 (local address 127.0.0.1:38214)
0924 16:02:53.814037 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:53.814093 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:53.814217 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:53.814996 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:53.815001 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:53.815021 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:53.815175 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:53.815183 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.816366 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.816370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:53.817892 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:53.817903 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.818911 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.818915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:53.819039 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:53.819080 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 16:02:53.819203 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 16:02:53.819330 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":14,"thread_start_us":86,"threads_started":1}
I20260924 16:02:53.820086 4413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:53.813799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38214 (local address 127.25.109.2:46649)
0924 16:02:53.814114 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:53.814126 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:53.814170 (+ 44us) server_negotiation.cc:407] Connection header received
0924 16:02:53.814396 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:53.814401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:53.814627 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:53.814789 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:53.815313 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.816183 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.818027 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.818790 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.818913 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:53.819068 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:53.819808 (+ 740us) server_negotiation.cc:299] Negotiation successful
0924 16:02:53.819908 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":102,"threads_started":1}
I20260924 16:02:53.820498 4417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:53.815274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34222 (local address 127.25.109.1:37911)
0924 16:02:53.815669 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:53.815684 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:53.815703 (+ 19us) server_negotiation.cc:407] Connection header received
0924 16:02:53.815774 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:53.815779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:53.815961 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:53.816113 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:53.816653 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.817428 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.819144 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.819828 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.819965 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:53.820115 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:53.820210 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 16:02:53.820315 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":117,"threads_started":1}
I20260924 16:02:53.820550 4414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:53.814449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:35991 (local address 127.0.0.1:50326)
0924 16:02:53.814718 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:53.814734 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:53.814814 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:53.816115 (+ 1301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:53.816119 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:53.816138 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:53.816242 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:53.816252 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.817701 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.817705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:53.819063 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:53.819074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.820132 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.820138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:53.820234 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:53.820258 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 16:02:53.820308 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 16:02:53.820369 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":86,"thread_start_us":128,"threads_started":1}
I20260924 16:02:53.821224 4415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:53.814631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50326 (local address 127.25.109.3:35991)
0924 16:02:53.815662 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:53.815676 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:53.815700 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:02:53.815774 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:53.815779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:53.815961 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:53.816113 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:53.816366 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.817556 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.819169 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.819986 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.820760 (+ 774us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:53.820914 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:53.820967 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 16:02:53.821072 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":935,"thread_start_us":103,"threads_started":1}
I20260924 16:02:53.821720 4416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:53.814794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:37911 (local address 127.0.0.1:34222)
0924 16:02:53.815330 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:53.815343 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:53.815431 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:53.816367 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:53.816370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:53.816386 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:53.816539 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:53.816544 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.817847 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.817850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:53.818981 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:53.818993 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.819967 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.819971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:53.820065 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:53.820089 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 16:02:53.821512 (+ 1423us) client_negotiation.cc:240] Negotiation successful
0924 16:02:53.821597 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":9,"thread_start_us":83,"threads_started":1}
I20260924 16:02:53.823339 4200 tablet_service.cc:1467] Processing CreateTablet for tablet d88fd799b7604332a5b3eed0cf9d509d (DEFAULT_TABLE table=test-table [id=ab91ed86cdd34a069389be162d2dce4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:53.824465 4200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d88fd799b7604332a5b3eed0cf9d509d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:53.824581 4335 tablet_service.cc:1467] Processing CreateTablet for tablet d88fd799b7604332a5b3eed0cf9d509d (DEFAULT_TABLE table=test-table [id=ab91ed86cdd34a069389be162d2dce4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:53.824581 4065 tablet_service.cc:1467] Processing CreateTablet for tablet d88fd799b7604332a5b3eed0cf9d509d (DEFAULT_TABLE table=test-table [id=ab91ed86cdd34a069389be162d2dce4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:02:53.825621 4065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d88fd799b7604332a5b3eed0cf9d509d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:53.825642 4335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d88fd799b7604332a5b3eed0cf9d509d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:02:53.831290 4418 tablet_bootstrap.cc:492] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35: Bootstrap starting.
I20260924 16:02:53.832803 4419 tablet_bootstrap.cc:492] T d88fd799b7604332a5b3eed0cf9d509d P dff99776a0464b77b19bce7fdcc467b6: Bootstrap starting.
I20260924 16:02:53.834619 4418 tablet_bootstrap.cc:654] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:53.835474 4419 tablet_bootstrap.cc:654] T d88fd799b7604332a5b3eed0cf9d509d P dff99776a0464b77b19bce7fdcc467b6: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:53.835675 4418 log.cc:824] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:53.836328 4419 log.cc:824] T d88fd799b7604332a5b3eed0cf9d509d P dff99776a0464b77b19bce7fdcc467b6: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:53.836802 4420 tablet_bootstrap.cc:492] T d88fd799b7604332a5b3eed0cf9d509d P d77bfffcb5ae49eda017aa7874898da9: Bootstrap starting.
I20260924 16:02:53.837996 4418 tablet_bootstrap.cc:492] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35: No bootstrap required, opened a new log
I20260924 16:02:53.838181 4419 tablet_bootstrap.cc:492] T d88fd799b7604332a5b3eed0cf9d509d P dff99776a0464b77b19bce7fdcc467b6: No bootstrap required, opened a new log
I20260924 16:02:53.838202 4418 ts_tablet_manager.cc:1397] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 16:02:53.838380 4419 ts_tablet_manager.cc:1397] T d88fd799b7604332a5b3eed0cf9d509d P dff99776a0464b77b19bce7fdcc467b6: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 16:02:53.839031 4420 tablet_bootstrap.cc:654] T d88fd799b7604332a5b3eed0cf9d509d P d77bfffcb5ae49eda017aa7874898da9: Neither blocks nor log segments found. Creating new log.
I20260924 16:02:53.840260 4420 log.cc:824] T d88fd799b7604332a5b3eed0cf9d509d P d77bfffcb5ae49eda017aa7874898da9: Log is configured to *not* fsync() on all Append() calls
I20260924 16:02:53.841641 4419 raft_consensus.cc:348] T d88fd799b7604332a5b3eed0cf9d509d P dff99776a0464b77b19bce7fdcc467b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c11e510346784980a5ff729253e0de35" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46649 } } peers { permanent_uuid: "dff99776a0464b77b19bce7fdcc467b6" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35991 } } peers { permanent_uuid: "d77bfffcb5ae49eda017aa7874898da9" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37911 } }
I20260924 16:02:53.841902 4419 raft_consensus.cc:374] T d88fd799b7604332a5b3eed0cf9d509d P dff99776a0464b77b19bce7fdcc467b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:53.841953 4419 raft_consensus.cc:729] T d88fd799b7604332a5b3eed0cf9d509d P dff99776a0464b77b19bce7fdcc467b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff99776a0464b77b19bce7fdcc467b6, State: Initialized, Role: FOLLOWER
I20260924 16:02:53.842458 4420 tablet_bootstrap.cc:492] T d88fd799b7604332a5b3eed0cf9d509d P d77bfffcb5ae49eda017aa7874898da9: No bootstrap required, opened a new log
I20260924 16:02:53.842422 4419 consensus_queue.cc:260] T d88fd799b7604332a5b3eed0cf9d509d P dff99776a0464b77b19bce7fdcc467b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c11e510346784980a5ff729253e0de35" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46649 } } peers { permanent_uuid: "dff99776a0464b77b19bce7fdcc467b6" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35991 } } peers { permanent_uuid: "d77bfffcb5ae49eda017aa7874898da9" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37911 } }
I20260924 16:02:53.842186 4418 raft_consensus.cc:348] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c11e510346784980a5ff729253e0de35" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46649 } } peers { permanent_uuid: "dff99776a0464b77b19bce7fdcc467b6" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35991 } } peers { permanent_uuid: "d77bfffcb5ae49eda017aa7874898da9" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37911 } }
I20260924 16:02:53.842726 4418 raft_consensus.cc:374] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:53.842784 4418 raft_consensus.cc:729] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c11e510346784980a5ff729253e0de35, State: Initialized, Role: FOLLOWER
I20260924 16:02:53.842876 4420 ts_tablet_manager.cc:1397] T d88fd799b7604332a5b3eed0cf9d509d P d77bfffcb5ae49eda017aa7874898da9: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260924 16:02:53.843274 4418 consensus_queue.cc:260] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c11e510346784980a5ff729253e0de35" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46649 } } peers { permanent_uuid: "dff99776a0464b77b19bce7fdcc467b6" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35991 } } peers { permanent_uuid: "d77bfffcb5ae49eda017aa7874898da9" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37911 } }
I20260924 16:02:53.843586 4419 ts_tablet_manager.cc:1428] T d88fd799b7604332a5b3eed0cf9d509d P dff99776a0464b77b19bce7fdcc467b6: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260924 16:02:53.844058 4401 heartbeater.cc:503] Master 127.25.109.62:33653 was elected leader, sending a full tablet report...
I20260924 16:02:53.844158 4267 heartbeater.cc:503] Master 127.25.109.62:33653 was elected leader, sending a full tablet report...
I20260924 16:02:53.844241 4418 ts_tablet_manager.cc:1428] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 16:02:53.846676 4420 raft_consensus.cc:348] T d88fd799b7604332a5b3eed0cf9d509d P d77bfffcb5ae49eda017aa7874898da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c11e510346784980a5ff729253e0de35" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46649 } } peers { permanent_uuid: "dff99776a0464b77b19bce7fdcc467b6" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35991 } } peers { permanent_uuid: "d77bfffcb5ae49eda017aa7874898da9" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37911 } }
I20260924 16:02:53.846969 4420 raft_consensus.cc:374] T d88fd799b7604332a5b3eed0cf9d509d P d77bfffcb5ae49eda017aa7874898da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:02:53.847038 4420 raft_consensus.cc:729] T d88fd799b7604332a5b3eed0cf9d509d P d77bfffcb5ae49eda017aa7874898da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d77bfffcb5ae49eda017aa7874898da9, State: Initialized, Role: FOLLOWER
I20260924 16:02:53.847471 4420 consensus_queue.cc:260] T d88fd799b7604332a5b3eed0cf9d509d P d77bfffcb5ae49eda017aa7874898da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c11e510346784980a5ff729253e0de35" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46649 } } peers { permanent_uuid: "dff99776a0464b77b19bce7fdcc467b6" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35991 } } peers { permanent_uuid: "d77bfffcb5ae49eda017aa7874898da9" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37911 } }
I20260924 16:02:53.848156 4132 heartbeater.cc:503] Master 127.25.109.62:33653 was elected leader, sending a full tablet report...
I20260924 16:02:53.848534 4420 ts_tablet_manager.cc:1428] T d88fd799b7604332a5b3eed0cf9d509d P d77bfffcb5ae49eda017aa7874898da9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 16:02:53.852373 4425 raft_consensus.cc:482] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:02:53.852594 4425 raft_consensus.cc:504] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c11e510346784980a5ff729253e0de35" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46649 } } peers { permanent_uuid: "dff99776a0464b77b19bce7fdcc467b6" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35991 } } peers { permanent_uuid: "d77bfffcb5ae49eda017aa7874898da9" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37911 } }
I20260924 16:02:53.853622 4425 leader_election.cc:288] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff99776a0464b77b19bce7fdcc467b6 (127.25.109.3:35991), d77bfffcb5ae49eda017aa7874898da9 (127.25.109.1:37911)
I20260924 16:02:53.857839 4415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:53.853918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:52593 (local address 127.25.109.3:35991)
0924 16:02:53.854049 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:53.854056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:53.854075 (+ 19us) server_negotiation.cc:407] Connection header received
0924 16:02:53.854170 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:53.854173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:53.854227 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:53.854313 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:53.854626 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.855367 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.856768 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.857254 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.857349 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:53.857391 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:53.857623 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 16:02:53.857701 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 16:02:53.857841 4269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:53.853863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:35991 (local address 127.25.109.2:52593)
0924 16:02:53.853969 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:53.853986 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:53.854073 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:53.854357 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:53.854359 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:53.854378 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:53.854513 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:53.854518 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.855466 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.855469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:53.856648 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:53.856658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.857369 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.857373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:53.857492 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:53.857518 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 16:02:53.857591 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 16:02:53.857685 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":30}
I20260924 16:02:53.857982 4417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:53.854209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:50235 (local address 127.25.109.1:37911)
0924 16:02:53.854346 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:53.854351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:53.854711 (+ 360us) server_negotiation.cc:407] Connection header received
0924 16:02:53.854809 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:53.854811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:53.854851 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:53.854906 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:02:53.855173 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.855804 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.856972 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.857412 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.857531 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:53.857598 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:53.857787 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 16:02:53.857867 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 16:02:53.858076 4427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:53.854114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:37911 (local address 127.25.109.2:50235)
0924 16:02:53.854596 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:53.854631 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 16:02:53.854730 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:02:53.854940 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:02:53.854943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:02:53.854960 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:02:53.855066 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:53.855072 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.855905 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.855907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:53.856841 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:02:53.856854 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.857537 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.857540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:02:53.857657 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:53.857681 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 16:02:53.857757 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 16:02:53.857822 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":18,"thread_start_us":145,"threads_started":1}
I20260924 16:02:53.858791 4085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d88fd799b7604332a5b3eed0cf9d509d" candidate_uuid: "c11e510346784980a5ff729253e0de35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d77bfffcb5ae49eda017aa7874898da9" is_pre_election: true
I20260924 16:02:53.858793 4355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d88fd799b7604332a5b3eed0cf9d509d" candidate_uuid: "c11e510346784980a5ff729253e0de35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff99776a0464b77b19bce7fdcc467b6" is_pre_election: true
I20260924 16:02:53.859108 4355 raft_consensus.cc:2454] T d88fd799b7604332a5b3eed0cf9d509d P dff99776a0464b77b19bce7fdcc467b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c11e510346784980a5ff729253e0de35 in term 0.
I20260924 16:02:53.859108 4085 raft_consensus.cc:2454] T d88fd799b7604332a5b3eed0cf9d509d P d77bfffcb5ae49eda017aa7874898da9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c11e510346784980a5ff729253e0de35 in term 0.
I20260924 16:02:53.859572 4155 leader_election.cc:302] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c11e510346784980a5ff729253e0de35, d77bfffcb5ae49eda017aa7874898da9; no voters:
I20260924 16:02:53.859807 4425 raft_consensus.cc:2781] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:02:53.859910 4425 raft_consensus.cc:482] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:02:53.859985 4425 raft_consensus.cc:3037] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:53.861212 4425 raft_consensus.cc:504] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c11e510346784980a5ff729253e0de35" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46649 } } peers { permanent_uuid: "dff99776a0464b77b19bce7fdcc467b6" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35991 } } peers { permanent_uuid: "d77bfffcb5ae49eda017aa7874898da9" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37911 } }
I20260924 16:02:53.861624 4425 leader_election.cc:288] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35 [CANDIDATE]: Term 1 election: Requested vote from peers dff99776a0464b77b19bce7fdcc467b6 (127.25.109.3:35991), d77bfffcb5ae49eda017aa7874898da9 (127.25.109.1:37911)
I20260924 16:02:53.862083 4355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d88fd799b7604332a5b3eed0cf9d509d" candidate_uuid: "c11e510346784980a5ff729253e0de35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff99776a0464b77b19bce7fdcc467b6"
I20260924 16:02:53.862092 4085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d88fd799b7604332a5b3eed0cf9d509d" candidate_uuid: "c11e510346784980a5ff729253e0de35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d77bfffcb5ae49eda017aa7874898da9"
I20260924 16:02:53.862196 4355 raft_consensus.cc:3037] T d88fd799b7604332a5b3eed0cf9d509d P dff99776a0464b77b19bce7fdcc467b6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:53.862228 4085 raft_consensus.cc:3037] T d88fd799b7604332a5b3eed0cf9d509d P d77bfffcb5ae49eda017aa7874898da9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:02:53.863255 4085 raft_consensus.cc:2454] T d88fd799b7604332a5b3eed0cf9d509d P d77bfffcb5ae49eda017aa7874898da9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c11e510346784980a5ff729253e0de35 in term 1.
I20260924 16:02:53.863263 4355 raft_consensus.cc:2454] T d88fd799b7604332a5b3eed0cf9d509d P dff99776a0464b77b19bce7fdcc467b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c11e510346784980a5ff729253e0de35 in term 1.
I20260924 16:02:53.863561 4155 leader_election.cc:302] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c11e510346784980a5ff729253e0de35, d77bfffcb5ae49eda017aa7874898da9; no voters:
I20260924 16:02:53.863852 4425 raft_consensus.cc:2781] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:02:53.864087 4425 raft_consensus.cc:686] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35 [term 1 LEADER]: Becoming Leader. State: Replica: c11e510346784980a5ff729253e0de35, State: Running, Role: LEADER
I20260924 16:02:53.864480 4425 consensus_queue.cc:237] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c11e510346784980a5ff729253e0de35" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46649 } } peers { permanent_uuid: "dff99776a0464b77b19bce7fdcc467b6" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35991 } } peers { permanent_uuid: "d77bfffcb5ae49eda017aa7874898da9" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37911 } }
I20260924 16:02:53.867806 3940 catalog_manager.cc:5495] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35 reported cstate change: term changed from 0 to 1, leader changed from <none> to c11e510346784980a5ff729253e0de35 (127.25.109.2). New cstate: current_term: 1 leader_uuid: "c11e510346784980a5ff729253e0de35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c11e510346784980a5ff729253e0de35" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dff99776a0464b77b19bce7fdcc467b6" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d77bfffcb5ae49eda017aa7874898da9" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37911 } health_report { overall_health: UNKNOWN } } }
I20260924 16:02:53.884975 4413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:53.881545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38228 (local address 127.25.109.2:46649)
0924 16:02:53.881677 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 16:02:53.881681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:02:53.881694 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:02:53.881821 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:02:53.881824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:02:53.881880 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:02:53.881985 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:02:53.882204 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.882705 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.883516 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:02:53.883724 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:02:53.883806 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:02:53.884064 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:02:53.884558 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:02:53.884653 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 16:02:53.884707 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 16:02:53.884756 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 16:02:53.891129 4355 raft_consensus.cc:1261] T d88fd799b7604332a5b3eed0cf9d509d P dff99776a0464b77b19bce7fdcc467b6 [term 1 FOLLOWER]: Refusing update from remote peer c11e510346784980a5ff729253e0de35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:02:53.891176 4085 raft_consensus.cc:1261] T d88fd799b7604332a5b3eed0cf9d509d P d77bfffcb5ae49eda017aa7874898da9 [term 1 FOLLOWER]: Refusing update from remote peer c11e510346784980a5ff729253e0de35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:02:53.891975 4425 consensus_queue.cc:1035] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff99776a0464b77b19bce7fdcc467b6" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:53.892210 4428 consensus_queue.cc:1035] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d77bfffcb5ae49eda017aa7874898da9" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:02:53.900949 4432 mvcc.cc:205] Tried to move back new op lower bound from 7332928609850253312 to 7332928609751490560. Current Snapshot: MvccSnapshot[applied={T|T < 7332928609850253312}]
I20260924 16:02:53.908023 3940 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:02:53.908210 3940 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:02:53.910799 3940 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 955c1bd155954d4892b6f778b02b44e2: Sending DeleteTablet for 3 replicas of tablet d88fd799b7604332a5b3eed0cf9d509d
I20260924 16:02:53.911623 4200 tablet_service.cc:1514] Processing DeleteTablet for tablet d88fd799b7604332a5b3eed0cf9d509d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:53 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:38214
I20260924 16:02:53.911713 4335 tablet_service.cc:1514] Processing DeleteTablet for tablet d88fd799b7604332a5b3eed0cf9d509d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:53 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:50326
I20260924 16:02:53.911818 4065 tablet_service.cc:1514] Processing DeleteTablet for tablet d88fd799b7604332a5b3eed0cf9d509d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:02:53 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:34222
I20260924 16:02:53.912288 4442 tablet_replica.cc:330] T d88fd799b7604332a5b3eed0cf9d509d P d77bfffcb5ae49eda017aa7874898da9: stopping tablet replica
I20260924 16:02:53.912297 4440 tablet_replica.cc:330] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35: stopping tablet replica
I20260924 16:02:53.912652 4442 raft_consensus.cc:2229] T d88fd799b7604332a5b3eed0cf9d509d P d77bfffcb5ae49eda017aa7874898da9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:53.912688 4440 raft_consensus.cc:2229] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:02:53.912976 4442 raft_consensus.cc:2258] T d88fd799b7604332a5b3eed0cf9d509d P d77bfffcb5ae49eda017aa7874898da9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:53.913017 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 4002
I20260924 16:02:53.914232 4440 raft_consensus.cc:2258] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:53.914630 4441 tablet_replica.cc:330] T d88fd799b7604332a5b3eed0cf9d509d P dff99776a0464b77b19bce7fdcc467b6: stopping tablet replica
I20260924 16:02:53.914943 4441 raft_consensus.cc:2229] T d88fd799b7604332a5b3eed0cf9d509d P dff99776a0464b77b19bce7fdcc467b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:02:53.915249 4441 raft_consensus.cc:2258] T d88fd799b7604332a5b3eed0cf9d509d P dff99776a0464b77b19bce7fdcc467b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:02:53.916944 4440 ts_tablet_manager.cc:1905] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:53.920606 4440 ts_tablet_manager.cc:1918] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:02:53.920711 4440 log.cc:1196] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-1/wal/wals/d88fd799b7604332a5b3eed0cf9d509d
I20260924 16:02:53.921049 4440 ts_tablet_manager.cc:1939] T d88fd799b7604332a5b3eed0cf9d509d P c11e510346784980a5ff729253e0de35: Deleting consensus metadata
I20260924 16:02:53.921988 4441 ts_tablet_manager.cc:1905] T d88fd799b7604332a5b3eed0cf9d509d P dff99776a0464b77b19bce7fdcc467b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:02:53.922380 3926 catalog_manager.cc:4841] TS c11e510346784980a5ff729253e0de35 (127.25.109.2:46649): tablet d88fd799b7604332a5b3eed0cf9d509d (table test-table [id=ab91ed86cdd34a069389be162d2dce4d]) successfully deleted
W20260924 16:02:53.922714 3926 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.109.1:37911 (error 108)
I20260924 16:02:53.922892 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 4137
I20260924 16:02:53.924338 4416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:02:53.923958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:37911 (local address 127.0.0.1:34228)
0924 16:02:53.924154 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 16:02:53.924231 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.1:37911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260924 16:02:53.924543 3926 catalog_manager.cc:4584] TS d77bfffcb5ae49eda017aa7874898da9 (127.25.109.1:37911): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d88fd799b7604332a5b3eed0cf9d509d: Network error: Client connection negotiation failed: client connection to 127.25.109.1:37911: connect: Connection refused (error 111)
I20260924 16:02:53.926338 4441 ts_tablet_manager.cc:1918] T d88fd799b7604332a5b3eed0cf9d509d P dff99776a0464b77b19bce7fdcc467b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:02:53.926436 4441 log.cc:1196] T d88fd799b7604332a5b3eed0cf9d509d P dff99776a0464b77b19bce7fdcc467b6: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265708917108-26036-0/minicluster-data/ts-2/wal/wals/d88fd799b7604332a5b3eed0cf9d509d
I20260924 16:02:53.926750 4441 ts_tablet_manager.cc:1939] T d88fd799b7604332a5b3eed0cf9d509d P dff99776a0464b77b19bce7fdcc467b6: Deleting consensus metadata
I20260924 16:02:53.928020 3926 catalog_manager.cc:4841] TS dff99776a0464b77b19bce7fdcc467b6 (127.25.109.3:35991): tablet d88fd799b7604332a5b3eed0cf9d509d (table test-table [id=ab91ed86cdd34a069389be162d2dce4d]) successfully deleted
I20260924 16:02:53.930277 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 4272
I20260924 16:02:53.936045 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 3908
2026-09-24T16:02:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2256 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:02:53 dist-test-slave-n106 krb5kdc[4446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:02:53 dist-test-slave-n106 krb5kdc[4446](info): set up 2 sockets
Sep 24 16:02:53 dist-test-slave-n106 krb5kdc[4446](info): commencing operation
krb5kdc: starting...
W20260924 16:02:54.982486 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:02:55 dist-test-slave-n106 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:02:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:02:55Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.25.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.25.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.25.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/rangeradmin_127.25.109.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.25.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/rangeradmin_127.25.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.25.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.25.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.25.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/rangerlookup_127.25.109.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.25.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/rangerlookup_127.25.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.25.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/HTTP_127.25.109.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.25.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/HTTP_127.25.109.20@KRBTEST.COM.keytab.
I20260924 16:02:55.155143 26036 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskCcKW0d/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/postgres -l logfile start
2026-09-24 16:02:57.740 UTC [4476] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 16:02:57.740 UTC [4476] LOG: listening on IPv4 address "127.25.109.20", port 56091
2026-09-24 16:02:57.744 UTC [4476] LOG: listening on Unix socket "/tmp/.s.PGSQL.56091"
2026-09-24 16:02:57.778 UTC [4479] LOG: database system was shut down at 2026-09-24 16:02:56 UTC
2026-09-24 16:02:57.783 UTC [4476] LOG: database system is ready to accept connections
I20260924 16:02:58.817989 26036 mini_postgres.cc:96] Postgres bound to 56091
2026-09-24 16:02:58.826 UTC [4487] FATAL: database "slave" does not exist
127.25.109.20:56091 - accepting connections
I20260924 16:02:58.827090 26036 mini_ranger.cc:161] Starting Ranger...
I20260924 16:02:58.842545 26036 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 16:02:59.314764 26036 mini_ranger.cc:87] Created ranger Postgres database
I20260924 16:02:59.314863 26036 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ranger-admin
2026-09-24 16:02:59,739 [I] DB FLAVOR :POSTGRES
2026-09-24 16:02:59,739 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 16:02:59,739 [I] Checking connection..
2026-09-24 16:02:59,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:56091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 16:03:00,034 [I] Checking connection passed.
2026-09-24 16:03:00,035 [I] --------- Verifying version history table ---------
2026-09-24 16:03:00,035 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:56091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 16:03:00,300 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 16:03:00,301 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:56091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 16:03:00,551 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 16:03:00,552 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 16:03:00,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:56091/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 16:03:00.781 UTC [4551] WARNING: there is no transaction in progress
2026-09-24 16:03:00,788 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 16:03:00,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:56091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 16:03:01,051 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 16:03:01,051 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 16:03:01,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:56091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 16:03:01,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:56091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 16:03:01,541 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:56091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-n106.c.gcp-upstream.internal','N') ;"
2026-09-24 16:03:01,797 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:56091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 16:03:02,040 [I] Table x_portal_user does not exist in database ranger
2026-09-24 16:03:02,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:56091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 16:03:02,308 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 16:03:02,308 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:56091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 16:03:02,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:56091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 16:03:02,794 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 16:03:02,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:56091/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 16:03:03.754 UTC [4659] WARNING: there is no transaction in progress
2026-09-24 16:03:03.770 UTC [4659] WARNING: there is no transaction in progress
2026-09-24 16:03:03.785 UTC [4659] WARNING: there is no transaction in progress
2026-09-24 16:03:03.800 UTC [4659] WARNING: there is no transaction in progress
2026-09-24 16:03:03.816 UTC [4659] WARNING: there is no transaction in progress
2026-09-24 16:03:03.834 UTC [4659] WARNING: there is no transaction in progress
2026-09-24 16:03:03.885 UTC [4659] WARNING: there is no transaction in progress
2026-09-24 16:03:03.892 UTC [4659] WARNING: there is no transaction in progress
2026-09-24 16:03:03.906 UTC [4659] WARNING: there is no transaction in progress
2026-09-24 16:03:03.916 UTC [4659] WARNING: there is no transaction in progress
2026-09-24 16:03:03.926 UTC [4659] WARNING: there is no transaction in progress
2026-09-24 16:03:03.940 UTC [4659] WARNING: there is no transaction in progress
2026-09-24 16:03:03.950 UTC [4659] WARNING: there is no transaction in progress
2026-09-24 16:03:03.956 UTC [4659] WARNING: there is no transaction in progress
2026-09-24 16:03:03.962 UTC [4659] WARNING: there is no transaction in progress
2026-09-24 16:03:03.969 UTC [4659] WARNING: there is no transaction in progress
2026-09-24 16:03:04,578 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 16:03:04,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:56091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 16:03:04,830 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 16:03:04,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:56091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 16:03:05,091 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 16:03:05,091 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:56091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 16:03:05,351 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 16:03:05,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:56091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 16:03:05,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:56091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 16:03:05,858 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:56091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-n106.c.gcp-upstream.internal';"
2026-09-24 16:03:06,087 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 16:03:06,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:56091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 16:03:06,339 [I] DB_PATCHES have already been applied
I20260924 16:03:06.344990 26036 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ranger-admin:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskCcKW0d/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 16:03:06.345057 26036 mini_ranger.cc:193] Using host: 127.25.109.20
I20260924 16:03:06.348568 26036 mini_ranger.cc:227] Ranger admin URL: http://127.25.109.20:38129
Sep 24, 2026 4:03:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:03:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:03:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 4:03:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 4:03:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 4:03:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 4:03:07 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 16:03:08.615499 26036 mini_ranger.cc:160] Time spent starting Ranger: real 9.788s user 0.002s sys 0.010s
Sep 24, 2026 4:03:29 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 4:03:29 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 4:03:29 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 4:03:30 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 4:03:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 4:03:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 4:03:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 4:03:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 4:03:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 4:03:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 4:03:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 4:03:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 4:03:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 4:03:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 4:03:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 4:03:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 4:03:30 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 4:03:30 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 16:03:31.456054 26036 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.25.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.25.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.25.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/rangerkms_127.25.109.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.25.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/rangerkms_127.25.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.25.109.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.109.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.109.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/HTTP_127.25.109.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.25.109.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/HTTP_127.25.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 16:03:31 dist-test-slave-n106 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265811, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 16:03:31.541056 26036 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 16:03:31.550309 26036 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 16:03:32.375038 26036 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 16:03:32.375118 26036 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ranger-kms
I20260924 16:03:32.375223 26036 mini_ranger_kms.cc:227] Using postgres at 127.25.109.20:56091
2026-09-24 16:03:32,953 [I] DB FLAVOR :POSTGRES
2026-09-24 16:03:32,953 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 16:03:32,953 [I] Checking connection
2026-09-24 16:03:32,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:56091/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 16:03:33,213 [I] connection success
2026-09-24 16:03:33,213 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 16:03:33,213 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 16:03:33,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:56091/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 16:03:33,475 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 16:03:33,476 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 16:03:33,476 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 16:03:33,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:56091/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 16:03:33,772 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 16:03:34.241189 26036 mini_ranger_kms.cc:310] Created kms service
I20260924 16:03:34.455545 26036 mini_ranger_kms.cc:326] Created kudu user
I20260924 16:03:34.517084 26036 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 16:03:34 dist-test-slave-n106 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265811, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.25.109.20@KRBTEST.COM
I20260924 16:03:34.987375 26036 mini_ranger_kms.cc:384] Added ranger policy
I20260924 16:03:34.987739 26036 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ranger-kms:/tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 16:03:34.987776 26036 mini_ranger_kms.cc:240] Using host: 127.25.109.20
I20260924 16:03:34.991602 26036 mini_ranger_kms.cc:276] Ranger KMS PID: 4843
I20260924 16:03:34.991765 26036 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.25.109.20:36461
Sep 24, 2026 4:03:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:03:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:03:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 4:03:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 4:03:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 4:03:36 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 24 16:03:37 dist-test-slave-n106 krb5kdc[4446](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790265817, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.109.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:03:37.245600 26036 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.705s user 0.028s sys 0.212s
I20260924 16:03:37.245712 26036 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 16:03:37.245870 26036 mini_ranger_kms.cc:401] 127.25.109.20:36461/kms/v1/keys
Sep 24 16:03:37 dist-test-slave-n106 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265817, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.109.20@KRBTEST.COM for HTTP/127.25.109.20@KRBTEST.COM
Sep 24 16:03:38 dist-test-slave-n106 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265817, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.109.20@KRBTEST.COM for HTTP/127.25.109.20@KRBTEST.COM
Sep 24 16:03:40 dist-test-slave-n106 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
I20260924 16:03:40.351857 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:39409
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:40769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:39409
--ranger_config_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.109.20:36461/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:03:40.475711 4893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:03:40.475999 4893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:03:40.476039 4893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:03:40.480367 4893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:03:40.480489 4893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:03:40.480513 4893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:03:40.480530 4893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:03:40.480546 4893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:03:40.486217 4893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:40769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/master-0/wal
--ranger_kms_url=127.25.109.20:36461/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:39409
--ranger_config_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:39409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:03:40.487385 4893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:03:40.488583 4893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:03:40.498850 4898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:03:40.498883 4899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:03:40.499066 4893 server_base.cc:1034] running on GCE node
W20260924 16:03:40.499122 4901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:03:40.499707 4893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:03:40.500999 4893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:03:40.502179 4893 hybrid_clock.cc:648] HybridClock initialized: now 1790265820502155 us; error 47 us; skew 500 ppm
Sep 24 16:03:40 dist-test-slave-n106 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265820, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:03:40.505411 4893 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:03:40.506521 4893 webserver.cc:460] Webserver started at http://127.25.109.62:44593/ using document root <none> and password file <none>
I20260924 16:03:40.507200 4893 fs_manager.cc:359] Metadata directory not provided
I20260924 16:03:40.507273 4893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:03:40.507519 4893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:03:40 dist-test-slave-n106 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265820, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for HTTP/127.25.109.20@KRBTEST.COM
I20260924 16:03:40.567935 4893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "32f354e4573a4d768fa2110829981ba2"
format_stamp: "Formatted at 2026-09-24 16:03:40 on dist-test-slave-n106"
server_key: "f44cb7dbf566f6458199336539346b86"
server_key_iv: "2fd433590af2b0e1be3f7a3b20f88bf6"
server_key_version: "kuduclusterkey@0"
I20260924 16:03:40.568655 4893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "32f354e4573a4d768fa2110829981ba2"
format_stamp: "Formatted at 2026-09-24 16:03:40 on dist-test-slave-n106"
server_key: "f44cb7dbf566f6458199336539346b86"
server_key_iv: "2fd433590af2b0e1be3f7a3b20f88bf6"
server_key_version: "kuduclusterkey@0"
I20260924 16:03:40.572994 4893 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:03:40.592777 4909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:40.594485 4893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:03:40.594635 4893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "32f354e4573a4d768fa2110829981ba2"
format_stamp: "Formatted at 2026-09-24 16:03:40 on dist-test-slave-n106"
server_key: "f44cb7dbf566f6458199336539346b86"
server_key_iv: "2fd433590af2b0e1be3f7a3b20f88bf6"
server_key_version: "kuduclusterkey@0"
I20260924 16:03:40.594785 4893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:03:40.610420 4893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:03:40.613775 4893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:03:40.613970 4893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:03:40.622874 4893 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:39409
I20260924 16:03:40.622884 4961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:39409 every 8 connection(s)
I20260924 16:03:40.624173 4893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:03:40.627269 4962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:03:40.630470 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 4893
I20260924 16:03:40.630609 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:03:40.630924 26036 external_mini_cluster.cc:1428] Setting key de669df1df4cdc6fabb3194f131e41ac
I20260924 16:03:40.634111 4962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32f354e4573a4d768fa2110829981ba2: Bootstrap starting.
I20260924 16:03:40.636940 4962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32f354e4573a4d768fa2110829981ba2: Neither blocks nor log segments found. Creating new log.
I20260924 16:03:40.637850 4962 log.cc:824] T 00000000000000000000000000000000 P 32f354e4573a4d768fa2110829981ba2: Log is configured to *not* fsync() on all Append() calls
I20260924 16:03:40.640343 4962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32f354e4573a4d768fa2110829981ba2: No bootstrap required, opened a new log
Sep 24 16:03:40 dist-test-slave-n106 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:03:40.645308 4962 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32f354e4573a4d768fa2110829981ba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32f354e4573a4d768fa2110829981ba2" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 39409 } }
I20260924 16:03:40.645715 4962 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32f354e4573a4d768fa2110829981ba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:03:40.645861 4962 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32f354e4573a4d768fa2110829981ba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32f354e4573a4d768fa2110829981ba2, State: Initialized, Role: FOLLOWER
I20260924 16:03:40.646410 4962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32f354e4573a4d768fa2110829981ba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32f354e4573a4d768fa2110829981ba2" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 39409 } }
I20260924 16:03:40.646584 4962 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32f354e4573a4d768fa2110829981ba2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:03:40.646680 4962 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32f354e4573a4d768fa2110829981ba2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:03:40.646792 4962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32f354e4573a4d768fa2110829981ba2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:03:40.646889 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:40.632635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42666 (local address 127.25.109.62:39409)
0924 16:03:40.633131 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:40.633152 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:40.633196 (+ 44us) server_negotiation.cc:407] Connection header received
0924 16:03:40.634042 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:40.634067 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:40.634509 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:40.634888 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:03:40.635765 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:40.637271 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:40.638582 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:40.639104 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:40.639296 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:40.641982 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:03:40.641988 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:03:40.642002 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:03:40.642032 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:03:40.643748 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:40.644231 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:40.644236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:40.644240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:40.644347 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:40.644657 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:40.644662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:40.644663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:40.645073 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:03:40.645297 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:40.645576 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 16:03:40.645893 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":147,"threads_started":1}
I20260924 16:03:40.649046 4962 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32f354e4573a4d768fa2110829981ba2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32f354e4573a4d768fa2110829981ba2" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 39409 } }
I20260924 16:03:40.649446 4962 leader_election.cc:302] T 00000000000000000000000000000000 P 32f354e4573a4d768fa2110829981ba2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32f354e4573a4d768fa2110829981ba2; no voters:
I20260924 16:03:40.649822 4962 leader_election.cc:288] T 00000000000000000000000000000000 P 32f354e4573a4d768fa2110829981ba2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:03:40.649901 4967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32f354e4573a4d768fa2110829981ba2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:03:40.650126 4967 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32f354e4573a4d768fa2110829981ba2 [term 1 LEADER]: Becoming Leader. State: Replica: 32f354e4573a4d768fa2110829981ba2, State: Running, Role: LEADER
I20260924 16:03:40.650490 4967 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32f354e4573a4d768fa2110829981ba2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32f354e4573a4d768fa2110829981ba2" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 39409 } }
I20260924 16:03:40.651202 4962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32f354e4573a4d768fa2110829981ba2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:03:40.652281 4968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32f354e4573a4d768fa2110829981ba2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32f354e4573a4d768fa2110829981ba2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32f354e4573a4d768fa2110829981ba2" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 39409 } } }
I20260924 16:03:40.652551 4969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32f354e4573a4d768fa2110829981ba2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32f354e4573a4d768fa2110829981ba2. Latest consensus state: current_term: 1 leader_uuid: "32f354e4573a4d768fa2110829981ba2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32f354e4573a4d768fa2110829981ba2" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 39409 } } }
I20260924 16:03:40.652643 4969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32f354e4573a4d768fa2110829981ba2 [sys.catalog]: This master's current role is: LEADER
I20260924 16:03:40.652935 4968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32f354e4573a4d768fa2110829981ba2 [sys.catalog]: This master's current role is: LEADER
I20260924 16:03:40.653282 4976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:03:40.653610 4962 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 16:03:40.660456 4976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:03:40.668283 4976 catalog_manager.cc:1347] Generated new cluster ID: ee1b3ad50d8b4b75b9cecae3f4f5e13e
I20260924 16:03:40.668350 4976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:03:40.683037 4976 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:03:40.684504 4976 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:03:40.693192 4976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32f354e4573a4d768fa2110829981ba2: Generated new TSK 0
I20260924 16:03:40.694155 4976 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
I20260924 16:03:41.388137 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.109.20:36461/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:0
--local_ip_for_outbound_sockets=127.25.109.1
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:39409
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:40769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:03:41.511025 5007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:03:41.511327 5007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:03:41.511399 5007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:03:41.517544 5007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:03:41.517627 5007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:03:41.517745 5007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:03:41.525871 5007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:40769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.25.109.20:36461/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:39409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:03:41.527091 5007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:03:41.528263 5007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:03:41.538820 5013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:03:41.538970 5007 server_base.cc:1034] running on GCE node
W20260924 16:03:41.539033 5012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:03:41.538851 5015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:03:41.539670 5007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:03:41.540323 5007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:03:41.541514 5007 hybrid_clock.cc:648] HybridClock initialized: now 1790265821541485 us; error 43 us; skew 500 ppm
Sep 24 16:03:41 dist-test-slave-n106 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265821, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:03:41.544587 5007 init.cc:377] Logged in from keytab as kudu/127.25.109.1@KRBTEST.COM (short username kudu)
I20260924 16:03:41.545677 5007 webserver.cc:460] Webserver started at http://127.25.109.1:42509/ using document root <none> and password file <none>
I20260924 16:03:41.546443 5007 fs_manager.cc:359] Metadata directory not provided
I20260924 16:03:41.546541 5007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:03:41.546805 5007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:03:41 dist-test-slave-n106 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265821, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for HTTP/127.25.109.20@KRBTEST.COM
I20260924 16:03:41.563597 5007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-0/data/instance:
uuid: "0ab8c81de7b44f32b8d8eb932872e58f"
format_stamp: "Formatted at 2026-09-24 16:03:41 on dist-test-slave-n106"
server_key: "3f1685083d261b67c0310a7f249dbd58"
server_key_iv: "d5ce57bd259e1360730072c7a61c84d8"
server_key_version: "kuduclusterkey@0"
I20260924 16:03:41.564308 5007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance:
uuid: "0ab8c81de7b44f32b8d8eb932872e58f"
format_stamp: "Formatted at 2026-09-24 16:03:41 on dist-test-slave-n106"
server_key: "3f1685083d261b67c0310a7f249dbd58"
server_key_iv: "d5ce57bd259e1360730072c7a61c84d8"
server_key_version: "kuduclusterkey@0"
I20260924 16:03:41.569911 5007 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 16:03:41.590286 5024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:41.592151 5007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 16:03:41.592288 5007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "0ab8c81de7b44f32b8d8eb932872e58f"
format_stamp: "Formatted at 2026-09-24 16:03:41 on dist-test-slave-n106"
server_key: "3f1685083d261b67c0310a7f249dbd58"
server_key_iv: "d5ce57bd259e1360730072c7a61c84d8"
server_key_version: "kuduclusterkey@0"
I20260924 16:03:41.592451 5007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:03:41.619110 5007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:03:41.622937 5007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:03:41.623142 5007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:03:41.623884 5007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:03:41.625023 5007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:03:41.625088 5007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:41.625144 5007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:03:41.625173 5007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:41.635907 5007 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:32807
I20260924 16:03:41.638087 5138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:32807 every 8 connection(s)
I20260924 16:03:41.638928 5007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:03:41.646807 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 5007
I20260924 16:03:41.646938 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance
I20260924 16:03:41.647257 26036 external_mini_cluster.cc:1428] Setting key 153caf22170c314dea1b20550eb79772
Sep 24 16:03:41 dist-test-slave-n106 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265821, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:03:41.658152 5141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:41.641566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:47807 (local address 127.25.109.62:39409)
0924 16:03:41.641869 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:41.641878 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:41.642849 (+ 971us) server_negotiation.cc:407] Connection header received
0924 16:03:41.645628 (+ 2779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:41.645633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:41.645697 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:41.645770 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:03:41.647437 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:41.648333 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:41.650090 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:41.650461 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:41.650538 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:41.653709 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:03:41.653714 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:03:41.653717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:03:41.653745 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:03:41.655663 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:41.656292 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:41.656296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:41.656298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:41.656395 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:41.656845 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:41.656849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:41.656851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:41.657076 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:03:41.657156 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:41.657838 (+ 682us) server_negotiation.cc:299] Negotiation successful
0924 16:03:41.657969 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":107,"threads_started":1}
I20260924 16:03:41.659925 5142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:41.642075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39409 (local address 127.25.109.1:47807)
0924 16:03:41.642668 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:41.642749 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0924 16:03:41.645360 (+ 2611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:03:41.646208 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:03:41.646217 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:03:41.646583 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:03:41.647250 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:41.647271 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:41.648539 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:41.648543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:41.649942 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:41.649955 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:41.650745 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:41.650749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:41.650953 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:41.651648 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:03:41.651672 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:03:41.653508 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:03:41.655812 (+ 2304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:03:41.655819 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:03:41.655836 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:03:41.656170 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:03:41.656548 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:03:41.656552 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:03:41.656554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:03:41.656734 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:03:41.657223 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:03:41.657230 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:03:41.657563 (+ 333us) client_negotiation.cc:769] Sending connection context
0924 16:03:41.658560 (+ 997us) client_negotiation.cc:240] Negotiation successful
0924 16:03:41.658899 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":132,"threads_started":1}
I20260924 16:03:41.661574 5139 heartbeater.cc:347] Connected to a master server at 127.25.109.62:39409
I20260924 16:03:41.661850 5139 heartbeater.cc:464] Registering TS with master...
I20260924 16:03:41.662424 5139 heartbeater.cc:511] Master 127.25.109.62:39409 requested a full tablet report, sending...
I20260924 16:03:41.664367 4926 ts_manager.cc:194] Registered new tserver with Master: 0ab8c81de7b44f32b8d8eb932872e58f (127.25.109.1:32807)
I20260924 16:03:41.665943 4926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.1@KRBTEST.COM'} at 127.25.109.1:47807
WARNING: no policy specified for kudu/127.25.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
I20260924 16:03:41.696226 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.109.20:36461/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:0
--local_ip_for_outbound_sockets=127.25.109.2
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:39409
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:40769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:03:41 dist-test-slave-n106 krb5kdc[4446](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790265821, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 16:03:41.867975 5147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:03:41.868431 5147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:03:41.868513 5147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:03:41.873626 5147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:03:41.873710 5147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:03:41.873799 5147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:03:41.882217 5147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:40769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.25.109.20:36461/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:39409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:03:41.883729 5147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:03:41.885138 5147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:03:41.901430 5154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:03:41.902151 5156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:03:41.902417 5153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:03:41.904037 5147 server_base.cc:1034] running on GCE node
I20260924 16:03:41.904677 5147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:03:41.905601 5147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:03:41.906805 5147 hybrid_clock.cc:648] HybridClock initialized: now 1790265821906763 us; error 54 us; skew 500 ppm
Sep 24 16:03:41 dist-test-slave-n106 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265821, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:03:41.910830 5147 init.cc:377] Logged in from keytab as kudu/127.25.109.2@KRBTEST.COM (short username kudu)
I20260924 16:03:41.912051 5147 webserver.cc:460] Webserver started at http://127.25.109.2:41413/ using document root <none> and password file <none>
I20260924 16:03:41.912854 5147 fs_manager.cc:359] Metadata directory not provided
I20260924 16:03:41.912946 5147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:03:41.913187 5147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:03:41 dist-test-slave-n106 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265821, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for HTTP/127.25.109.20@KRBTEST.COM
I20260924 16:03:41.939929 5147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-1/data/instance:
uuid: "9ed3085a243e4385bf7113043238a389"
format_stamp: "Formatted at 2026-09-24 16:03:41 on dist-test-slave-n106"
server_key: "f0c5b9861ade6f3cdfbcae35d8835c39"
server_key_iv: "39a34bc186cb3ef620a100baa5c13bdc"
server_key_version: "kuduclusterkey@0"
I20260924 16:03:41.940860 5147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance:
uuid: "9ed3085a243e4385bf7113043238a389"
format_stamp: "Formatted at 2026-09-24 16:03:41 on dist-test-slave-n106"
server_key: "f0c5b9861ade6f3cdfbcae35d8835c39"
server_key_iv: "39a34bc186cb3ef620a100baa5c13bdc"
server_key_version: "kuduclusterkey@0"
I20260924 16:03:41.947144 5147 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260924 16:03:41.971567 5163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:41.973405 5147 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s
I20260924 16:03:41.973572 5147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "9ed3085a243e4385bf7113043238a389"
format_stamp: "Formatted at 2026-09-24 16:03:41 on dist-test-slave-n106"
server_key: "f0c5b9861ade6f3cdfbcae35d8835c39"
server_key_iv: "39a34bc186cb3ef620a100baa5c13bdc"
server_key_version: "kuduclusterkey@0"
I20260924 16:03:41.973729 5147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:03:42.011636 5147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:03:42.015058 5147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:03:42.015283 5147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:03:42.015980 5147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:03:42.016978 5147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:03:42.017055 5147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:42.017130 5147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:03:42.017174 5147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:42.029151 5147 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:33377
I20260924 16:03:42.029203 5278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:33377 every 8 connection(s)
I20260924 16:03:42.030144 5147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:03:42.031730 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 5147
I20260924 16:03:42.031868 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance
I20260924 16:03:42.032199 26036 external_mini_cluster.cc:1428] Setting key daef93ac30f44516f596841ff2a97613
Sep 24 16:03:42 dist-test-slave-n106 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265821, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:03:42.047286 5141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:42.032446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:50387 (local address 127.25.109.62:39409)
0924 16:03:42.033877 (+ 1431us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:42.033882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:42.033899 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:03:42.034898 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:42.034902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:42.034959 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:42.035082 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:03:42.036312 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.037227 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.039871 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.040194 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.040304 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:42.043047 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:03:42.043051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:03:42.043053 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:03:42.043080 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:03:42.044727 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:42.045274 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:42.045278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:42.045280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:42.045364 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:42.045779 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:42.045783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:42.045785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:42.046053 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:03:42.046188 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:42.046882 (+ 694us) server_negotiation.cc:299] Negotiation successful
0924 16:03:42.047047 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1339}
WARNING: no policy specified for kudu/127.25.109.3@KRBTEST.COM; defaulting to no policy
I20260924 16:03:42.048243 5281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:42.032858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39409 (local address 127.25.109.2:50387)
0924 16:03:42.033388 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:42.033444 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 16:03:42.034638 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:03:42.035274 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:03:42.035281 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:03:42.035574 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:03:42.036138 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:42.036155 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.037402 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.037407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:42.039678 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:42.039701 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.040364 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.040369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:42.040608 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:42.041322 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:03:42.041339 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:03:42.042849 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:03:42.044852 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:03:42.044859 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:03:42.044876 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:03:42.045163 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:03:42.045461 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:03:42.045465 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:03:42.045467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:03:42.045656 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:03:42.046208 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:03:42.046215 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:03:42.046563 (+ 348us) client_negotiation.cc:769] Sending connection context
0924 16:03:42.046799 (+ 236us) client_negotiation.cc:240] Negotiation successful
0924 16:03:42.047160 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":141,"threads_started":1}
I20260924 16:03:42.052114 5279 heartbeater.cc:347] Connected to a master server at 127.25.109.62:39409
I20260924 16:03:42.052474 5279 heartbeater.cc:464] Registering TS with master...
I20260924 16:03:42.053306 5279 heartbeater.cc:511] Master 127.25.109.62:39409 requested a full tablet report, sending...
I20260924 16:03:42.054522 4926 ts_manager.cc:194] Registered new tserver with Master: 9ed3085a243e4385bf7113043238a389 (127.25.109.2:33377)
I20260924 16:03:42.055186 4926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.2@KRBTEST.COM'} at 127.25.109.2:50387
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
I20260924 16:03:42.075811 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.109.20:36461/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:0
--local_ip_for_outbound_sockets=127.25.109.3
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:39409
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:40769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:03:42 dist-test-slave-n106 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265821, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for HTTP/127.25.109.20@KRBTEST.COM
W20260924 16:03:42.253914 5284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:03:42.254307 5284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:03:42.254454 5284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:03:42.260613 5284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:03:42.260705 5284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:03:42.260805 5284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:03:42.269070 5284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:40769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.25.109.20:36461/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:39409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:03:42.270581 5284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:03:42.271975 5284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:03:42.282922 5290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:03:42.283384 5292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:03:42.282922 5289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:03:42.285651 5284 server_base.cc:1034] running on GCE node
I20260924 16:03:42.286091 5284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:03:42.286772 5284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:03:42.294135 5284 hybrid_clock.cc:648] HybridClock initialized: now 1790265822294091 us; error 52 us; skew 500 ppm
Sep 24 16:03:42 dist-test-slave-n106 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265822, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:03:42.297199 5284 init.cc:377] Logged in from keytab as kudu/127.25.109.3@KRBTEST.COM (short username kudu)
I20260924 16:03:42.298295 5284 webserver.cc:460] Webserver started at http://127.25.109.3:33501/ using document root <none> and password file <none>
I20260924 16:03:42.298962 5284 fs_manager.cc:359] Metadata directory not provided
I20260924 16:03:42.299074 5284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:03:42.299327 5284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:03:42 dist-test-slave-n106 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265822, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for HTTP/127.25.109.20@KRBTEST.COM
I20260924 16:03:42.325695 5284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-2/data/instance:
uuid: "08025e5eca1045a09f304b7a33df9364"
format_stamp: "Formatted at 2026-09-24 16:03:42 on dist-test-slave-n106"
server_key: "52049bc3b20f54dc1dec6b8e18e51a45"
server_key_iv: "32276e83d651903c9436f96efd113600"
server_key_version: "kuduclusterkey@0"
I20260924 16:03:42.326469 5284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance:
uuid: "08025e5eca1045a09f304b7a33df9364"
format_stamp: "Formatted at 2026-09-24 16:03:42 on dist-test-slave-n106"
server_key: "52049bc3b20f54dc1dec6b8e18e51a45"
server_key_iv: "32276e83d651903c9436f96efd113600"
server_key_version: "kuduclusterkey@0"
I20260924 16:03:42.332083 5284 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:03:42.349112 5299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:42.350667 5284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:03:42.350836 5284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "08025e5eca1045a09f304b7a33df9364"
format_stamp: "Formatted at 2026-09-24 16:03:42 on dist-test-slave-n106"
server_key: "52049bc3b20f54dc1dec6b8e18e51a45"
server_key_iv: "32276e83d651903c9436f96efd113600"
server_key_version: "kuduclusterkey@0"
I20260924 16:03:42.351029 5284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:03:42.373921 5284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:03:42.377789 5284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:03:42.378003 5284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:03:42.378793 5284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:03:42.379926 5284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:03:42.379995 5284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:42.380064 5284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:03:42.380095 5284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:03:42.391847 5284 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:44603
I20260924 16:03:42.391897 5413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:44603 every 8 connection(s)
I20260924 16:03:42.393131 5284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:03:42.399487 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 5284
I20260924 16:03:42.399618 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance
I20260924 16:03:42.399914 26036 external_mini_cluster.cc:1428] Setting key 782eb1e998257ef637c641a432cf306f
Sep 24 16:03:42 dist-test-slave-n106 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265822, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:03:42.411273 5141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:42.397174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:38095 (local address 127.25.109.62:39409)
0924 16:03:42.397321 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:42.397329 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:42.397348 (+ 19us) server_negotiation.cc:407] Connection header received
0924 16:03:42.398020 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:42.398024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:42.398113 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:42.398209 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:03:42.400952 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.401800 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.403476 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.403851 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.403980 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:42.406990 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:03:42.406994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:03:42.406997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:03:42.407027 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:03:42.408779 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:42.409422 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:42.409426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:42.409428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:42.409521 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:42.409970 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:42.409974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:42.409976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:42.410190 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:03:42.410284 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:42.410984 (+ 700us) server_negotiation.cc:299] Negotiation successful
0924 16:03:42.411109 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:03:42.412585 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:42.395990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:39409 (local address 127.25.109.3:38095)
0924 16:03:42.396631 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:42.396694 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 16:03:42.397757 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:03:42.398419 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:03:42.398428 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:03:42.398792 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:03:42.400775 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:42.400796 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.401961 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.401964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:42.403325 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:42.403336 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.404019 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.404023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:42.404226 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:42.404968 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:03:42.404989 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:03:42.406790 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:03:42.408906 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:03:42.408913 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:03:42.408930 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:03:42.409295 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:03:42.409643 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:03:42.409647 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:03:42.409649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:03:42.409853 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:03:42.410312 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:03:42.410319 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:03:42.410676 (+ 357us) client_negotiation.cc:769] Sending connection context
0924 16:03:42.410913 (+ 237us) client_negotiation.cc:240] Negotiation successful
0924 16:03:42.411279 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":147,"threads_started":1}
I20260924 16:03:42.414747 5414 heartbeater.cc:347] Connected to a master server at 127.25.109.62:39409
I20260924 16:03:42.415007 5414 heartbeater.cc:464] Registering TS with master...
I20260924 16:03:42.415575 5414 heartbeater.cc:511] Master 127.25.109.62:39409 requested a full tablet report, sending...
I20260924 16:03:42.416800 4926 ts_manager.cc:194] Registered new tserver with Master: 08025e5eca1045a09f304b7a33df9364 (127.25.109.3:44603)
I20260924 16:03:42.417611 4926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.3@KRBTEST.COM'} at 127.25.109.3:38095
I20260924 16:03:42.428783 26036 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:03:42.442377 5141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:42.432045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42670 (local address 127.25.109.62:39409)
0924 16:03:42.432220 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:42.432225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:42.432558 (+ 333us) server_negotiation.cc:407] Connection header received
0924 16:03:42.432711 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:42.432714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:42.432771 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:42.432865 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:03:42.433800 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.434621 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.435854 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.436197 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.436320 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:42.438171 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:03:42.438175 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:03:42.438178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:03:42.438210 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:03:42.440040 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:42.440595 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:42.440606 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:42.440608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:42.440709 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:03:42.441077 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:03:42.441080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:03:42.441082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:03:42.441332 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:03:42.441481 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:42.441805 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 16:03:42.442102 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 16:03:42.446880 4926 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:03:42.451696 4926 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:03:42.475879 5428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:42.465332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:33377 (local address 127.0.0.1:56264)
0924 16:03:42.468717 (+ 3385us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:42.468731 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:03:42.468821 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:03:42.470274 (+ 1453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:03:42.470277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:03:42.470297 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:03:42.470409 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:42.470415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.471871 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.471875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:42.473059 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:42.473070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.475273 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.475277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:42.475417 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:42.475466 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 16:03:42.475585 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 16:03:42.475711 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3323,"thread_start_us":97,"threads_started":1}
I20260924 16:03:42.476550 5431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:42.466095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56264 (local address 127.25.109.2:33377)
0924 16:03:42.468570 (+ 2475us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:42.468585 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:42.468911 (+ 326us) server_negotiation.cc:407] Connection header received
0924 16:03:42.469002 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:42.469007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:42.469185 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:42.469324 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:03:42.470540 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.471738 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.473226 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.474922 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.475042 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:42.475206 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:42.476261 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0924 16:03:42.476379 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2371,"thread_start_us":112,"threads_started":1}
I20260924 16:03:42.476913 5426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:42.463521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:44603 (local address 127.0.0.1:45282)
0924 16:03:42.464745 (+ 1224us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:42.464801 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 16:03:42.464940 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:03:42.467793 (+ 2853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:03:42.467801 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:03:42.467830 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:03:42.467981 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:42.467990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.469646 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.469650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:42.471134 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:42.471147 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.476520 (+ 5373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.476527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:42.476626 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:42.476652 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 16:03:42.476706 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 16:03:42.476775 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1015,"thread_start_us":99,"threads_started":1}
I20260924 16:03:42.477716 5427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:42.464397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45282 (local address 127.25.109.3:44603)
0924 16:03:42.467021 (+ 2624us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:42.467037 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:42.467061 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:03:42.467148 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:42.467154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:42.467377 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:42.467538 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:03:42.468148 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.469498 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.471315 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.472193 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.477235 (+ 5042us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:42.477393 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:42.477455 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 16:03:42.477567 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2449,"thread_start_us":144,"threads_started":1}
I20260924 16:03:42.478427 5429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:42.465102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:32807 (local address 127.0.0.1:45846)
0924 16:03:42.465824 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:42.465838 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:03:42.465932 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:03:42.472315 (+ 6383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:03:42.472318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:03:42.472340 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:03:42.472474 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:42.472483 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.473950 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.473953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:42.475182 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:42.475195 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.478004 (+ 2809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.478009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:42.478126 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:42.478151 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 16:03:42.478231 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 16:03:42.478303 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":90,"threads_started":1}
I20260924 16:03:42.479419 5430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:42.465730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45846 (local address 127.25.109.1:32807)
0924 16:03:42.467825 (+ 2095us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:42.467832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:42.467854 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:03:42.467930 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:42.467935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:42.468139 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:42.468278 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:03:42.472643 (+ 4365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.473807 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.475345 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.476186 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.476301 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:42.477935 (+ 1634us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:42.479163 (+ 1228us) server_negotiation.cc:299] Negotiation successful
0924 16:03:42.479274 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1990,"thread_start_us":89,"threads_started":1}
I20260924 16:03:42.481534 5212 tablet_service.cc:1467] Processing CreateTablet for tablet b83449b4193d4529bf4aa81d617a7f8f (DEFAULT_TABLE table=test-table [id=dcb3622ae5e1457da1b83d83bf2978de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:03:42.482798 5212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b83449b4193d4529bf4aa81d617a7f8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:03:42.482601 5070 tablet_service.cc:1467] Processing CreateTablet for tablet b83449b4193d4529bf4aa81d617a7f8f (DEFAULT_TABLE table=test-table [id=dcb3622ae5e1457da1b83d83bf2978de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:03:42.483664 5070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b83449b4193d4529bf4aa81d617a7f8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:03:42.486660 5347 tablet_service.cc:1467] Processing CreateTablet for tablet b83449b4193d4529bf4aa81d617a7f8f (DEFAULT_TABLE table=test-table [id=dcb3622ae5e1457da1b83d83bf2978de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:03:42.487640 5347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b83449b4193d4529bf4aa81d617a7f8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:03:42.506814 5433 tablet_bootstrap.cc:492] T b83449b4193d4529bf4aa81d617a7f8f P 08025e5eca1045a09f304b7a33df9364: Bootstrap starting.
I20260924 16:03:42.507753 5434 tablet_bootstrap.cc:492] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389: Bootstrap starting.
I20260924 16:03:42.510808 5432 tablet_bootstrap.cc:492] T b83449b4193d4529bf4aa81d617a7f8f P 0ab8c81de7b44f32b8d8eb932872e58f: Bootstrap starting.
I20260924 16:03:42.513365 5432 tablet_bootstrap.cc:654] T b83449b4193d4529bf4aa81d617a7f8f P 0ab8c81de7b44f32b8d8eb932872e58f: Neither blocks nor log segments found. Creating new log.
I20260924 16:03:42.515038 5433 tablet_bootstrap.cc:654] T b83449b4193d4529bf4aa81d617a7f8f P 08025e5eca1045a09f304b7a33df9364: Neither blocks nor log segments found. Creating new log.
I20260924 16:03:42.516117 5432 log.cc:824] T b83449b4193d4529bf4aa81d617a7f8f P 0ab8c81de7b44f32b8d8eb932872e58f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:03:42.517166 5433 log.cc:824] T b83449b4193d4529bf4aa81d617a7f8f P 08025e5eca1045a09f304b7a33df9364: Log is configured to *not* fsync() on all Append() calls
I20260924 16:03:42.519433 5434 tablet_bootstrap.cc:654] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389: Neither blocks nor log segments found. Creating new log.
I20260924 16:03:42.520327 5434 log.cc:824] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389: Log is configured to *not* fsync() on all Append() calls
I20260924 16:03:42.522181 5432 tablet_bootstrap.cc:492] T b83449b4193d4529bf4aa81d617a7f8f P 0ab8c81de7b44f32b8d8eb932872e58f: No bootstrap required, opened a new log
I20260924 16:03:42.522462 5432 ts_tablet_manager.cc:1397] T b83449b4193d4529bf4aa81d617a7f8f P 0ab8c81de7b44f32b8d8eb932872e58f: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260924 16:03:42.525892 5434 tablet_bootstrap.cc:492] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389: No bootstrap required, opened a new log
I20260924 16:03:42.526130 5434 ts_tablet_manager.cc:1397] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20260924 16:03:42.527292 5432 raft_consensus.cc:348] T b83449b4193d4529bf4aa81d617a7f8f P 0ab8c81de7b44f32b8d8eb932872e58f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08025e5eca1045a09f304b7a33df9364" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 44603 } } peers { permanent_uuid: "0ab8c81de7b44f32b8d8eb932872e58f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 32807 } } peers { permanent_uuid: "9ed3085a243e4385bf7113043238a389" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 33377 } }
I20260924 16:03:42.529481 5434 raft_consensus.cc:348] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08025e5eca1045a09f304b7a33df9364" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 44603 } } peers { permanent_uuid: "0ab8c81de7b44f32b8d8eb932872e58f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 32807 } } peers { permanent_uuid: "9ed3085a243e4385bf7113043238a389" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 33377 } }
I20260924 16:03:42.529788 5434 raft_consensus.cc:374] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:03:42.529819 5433 tablet_bootstrap.cc:492] T b83449b4193d4529bf4aa81d617a7f8f P 08025e5eca1045a09f304b7a33df9364: No bootstrap required, opened a new log
I20260924 16:03:42.529863 5434 raft_consensus.cc:729] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed3085a243e4385bf7113043238a389, State: Initialized, Role: FOLLOWER
I20260924 16:03:42.530150 5433 ts_tablet_manager.cc:1397] T b83449b4193d4529bf4aa81d617a7f8f P 08025e5eca1045a09f304b7a33df9364: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.000s
I20260924 16:03:42.530426 5434 consensus_queue.cc:260] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08025e5eca1045a09f304b7a33df9364" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 44603 } } peers { permanent_uuid: "0ab8c81de7b44f32b8d8eb932872e58f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 32807 } } peers { permanent_uuid: "9ed3085a243e4385bf7113043238a389" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 33377 } }
I20260924 16:03:42.531540 5434 ts_tablet_manager.cc:1428] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 16:03:42.531960 5279 heartbeater.cc:503] Master 127.25.109.62:39409 was elected leader, sending a full tablet report...
W20260924 16:03:42.534924 5280 tablet.cc:2369] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:03:42.535842 5433 raft_consensus.cc:348] T b83449b4193d4529bf4aa81d617a7f8f P 08025e5eca1045a09f304b7a33df9364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08025e5eca1045a09f304b7a33df9364" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 44603 } } peers { permanent_uuid: "0ab8c81de7b44f32b8d8eb932872e58f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 32807 } } peers { permanent_uuid: "9ed3085a243e4385bf7113043238a389" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 33377 } }
I20260924 16:03:42.536213 5433 raft_consensus.cc:374] T b83449b4193d4529bf4aa81d617a7f8f P 08025e5eca1045a09f304b7a33df9364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:03:42.536350 5433 raft_consensus.cc:729] T b83449b4193d4529bf4aa81d617a7f8f P 08025e5eca1045a09f304b7a33df9364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08025e5eca1045a09f304b7a33df9364, State: Initialized, Role: FOLLOWER
I20260924 16:03:42.536959 5433 consensus_queue.cc:260] T b83449b4193d4529bf4aa81d617a7f8f P 08025e5eca1045a09f304b7a33df9364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08025e5eca1045a09f304b7a33df9364" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 44603 } } peers { permanent_uuid: "0ab8c81de7b44f32b8d8eb932872e58f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 32807 } } peers { permanent_uuid: "9ed3085a243e4385bf7113043238a389" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 33377 } }
I20260924 16:03:42.537195 5432 raft_consensus.cc:374] T b83449b4193d4529bf4aa81d617a7f8f P 0ab8c81de7b44f32b8d8eb932872e58f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:03:42.537298 5432 raft_consensus.cc:729] T b83449b4193d4529bf4aa81d617a7f8f P 0ab8c81de7b44f32b8d8eb932872e58f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ab8c81de7b44f32b8d8eb932872e58f, State: Initialized, Role: FOLLOWER
I20260924 16:03:42.538554 5433 ts_tablet_manager.cc:1428] T b83449b4193d4529bf4aa81d617a7f8f P 08025e5eca1045a09f304b7a33df9364: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260924 16:03:42.538988 5432 consensus_queue.cc:260] T b83449b4193d4529bf4aa81d617a7f8f P 0ab8c81de7b44f32b8d8eb932872e58f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08025e5eca1045a09f304b7a33df9364" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 44603 } } peers { permanent_uuid: "0ab8c81de7b44f32b8d8eb932872e58f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 32807 } } peers { permanent_uuid: "9ed3085a243e4385bf7113043238a389" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 33377 } }
I20260924 16:03:42.540272 5432 ts_tablet_manager.cc:1428] T b83449b4193d4529bf4aa81d617a7f8f P 0ab8c81de7b44f32b8d8eb932872e58f: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20260924 16:03:42.540729 5414 heartbeater.cc:503] Master 127.25.109.62:39409 was elected leader, sending a full tablet report...
I20260924 16:03:42.542140 5139 heartbeater.cc:503] Master 127.25.109.62:39409 was elected leader, sending a full tablet report...
Sep 24 16:03:42 dist-test-slave-n106 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265821, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for HTTP/127.25.109.20@KRBTEST.COM
I20260924 16:03:42.552906 5439 raft_consensus.cc:482] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:03:42.553130 5439 raft_consensus.cc:504] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08025e5eca1045a09f304b7a33df9364" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 44603 } } peers { permanent_uuid: "0ab8c81de7b44f32b8d8eb932872e58f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 32807 } } peers { permanent_uuid: "9ed3085a243e4385bf7113043238a389" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 33377 } }
I20260924 16:03:42.559443 5439 leader_election.cc:288] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08025e5eca1045a09f304b7a33df9364 (127.25.109.3:44603), 0ab8c81de7b44f32b8d8eb932872e58f (127.25.109.1:32807)
I20260924 16:03:42.560827 5281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:42.555710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:44603 (local address 127.25.109.2:46651)
0924 16:03:42.556011 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:42.556026 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:03:42.556115 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:03:42.556868 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:03:42.556871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:03:42.556890 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:03:42.557006 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:42.557012 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.558070 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.558074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:42.559299 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:42.559311 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.560323 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.560327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:42.560477 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:42.560502 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 16:03:42.560578 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 16:03:42.560684 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":21}
I20260924 16:03:42.560925 5427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:42.556274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:46651 (local address 127.25.109.3:44603)
0924 16:03:42.556521 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:42.556526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:42.556542 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:03:42.556595 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:42.556598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:42.556647 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:42.556742 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:03:42.557142 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.557889 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.559447 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.560084 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.560203 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:42.560261 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:42.560716 (+ 455us) server_negotiation.cc:299] Negotiation successful
0924 16:03:42.560799 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260924 16:03:42.561914 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b83449b4193d4529bf4aa81d617a7f8f" candidate_uuid: "9ed3085a243e4385bf7113043238a389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08025e5eca1045a09f304b7a33df9364" is_pre_election: true
I20260924 16:03:42.562260 5349 raft_consensus.cc:2454] T b83449b4193d4529bf4aa81d617a7f8f P 08025e5eca1045a09f304b7a33df9364 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ed3085a243e4385bf7113043238a389 in term 0.
I20260924 16:03:42.562901 5166 leader_election.cc:302] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08025e5eca1045a09f304b7a33df9364, 9ed3085a243e4385bf7113043238a389; no voters:
I20260924 16:03:42.563230 5439 raft_consensus.cc:2781] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:03:42.563340 5439 raft_consensus.cc:482] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:03:42.563398 5439 raft_consensus.cc:3037] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:03:42.564884 5439 raft_consensus.cc:504] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08025e5eca1045a09f304b7a33df9364" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 44603 } } peers { permanent_uuid: "0ab8c81de7b44f32b8d8eb932872e58f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 32807 } } peers { permanent_uuid: "9ed3085a243e4385bf7113043238a389" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 33377 } }
I20260924 16:03:42.565367 5439 leader_election.cc:288] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389 [CANDIDATE]: Term 1 election: Requested vote from peers 08025e5eca1045a09f304b7a33df9364 (127.25.109.3:44603), 0ab8c81de7b44f32b8d8eb932872e58f (127.25.109.1:32807)
I20260924 16:03:42.565850 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b83449b4193d4529bf4aa81d617a7f8f" candidate_uuid: "9ed3085a243e4385bf7113043238a389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08025e5eca1045a09f304b7a33df9364"
I20260924 16:03:42.565985 5349 raft_consensus.cc:3037] T b83449b4193d4529bf4aa81d617a7f8f P 08025e5eca1045a09f304b7a33df9364 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:03:42.567303 5349 raft_consensus.cc:2454] T b83449b4193d4529bf4aa81d617a7f8f P 08025e5eca1045a09f304b7a33df9364 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ed3085a243e4385bf7113043238a389 in term 1.
I20260924 16:03:42.567705 5166 leader_election.cc:302] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08025e5eca1045a09f304b7a33df9364, 9ed3085a243e4385bf7113043238a389; no voters:
I20260924 16:03:42.567867 5430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:42.556264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:40419 (local address 127.25.109.1:32807)
0924 16:03:42.556383 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:42.556387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:42.560037 (+ 3650us) server_negotiation.cc:407] Connection header received
0924 16:03:42.560263 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:42.560267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:42.560318 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:42.560369 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:03:42.561693 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.562455 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.566241 (+ 3786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.566794 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.566906 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:42.566971 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:42.567645 (+ 674us) server_negotiation.cc:299] Negotiation successful
0924 16:03:42.567736 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 16:03:42.568094 5439 raft_consensus.cc:2781] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:03:42.568560 5439 raft_consensus.cc:686] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389 [term 1 LEADER]: Becoming Leader. State: Replica: 9ed3085a243e4385bf7113043238a389, State: Running, Role: LEADER
I20260924 16:03:42.568945 5439 consensus_queue.cc:237] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08025e5eca1045a09f304b7a33df9364" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 44603 } } peers { permanent_uuid: "0ab8c81de7b44f32b8d8eb932872e58f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 32807 } } peers { permanent_uuid: "9ed3085a243e4385bf7113043238a389" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 33377 } }
I20260924 16:03:42.572302 4925 catalog_manager.cc:5495] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ed3085a243e4385bf7113043238a389 (127.25.109.2). New cstate: current_term: 1 leader_uuid: "9ed3085a243e4385bf7113043238a389" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08025e5eca1045a09f304b7a33df9364" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 44603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ab8c81de7b44f32b8d8eb932872e58f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 32807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ed3085a243e4385bf7113043238a389" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 33377 } health_report { overall_health: HEALTHY } } }
I20260924 16:03:42.576545 5442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:42.555747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:32807 (local address 127.25.109.2:40419)
0924 16:03:42.559709 (+ 3962us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:42.559724 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:03:42.560127 (+ 403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:03:42.560541 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:03:42.560544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:03:42.560563 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:03:42.561559 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:42.561566 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.562630 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.562633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:42.566067 (+ 3434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:03:42.566082 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.567381 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.567384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:03:42.567484 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:42.567511 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 16:03:42.576234 (+ 8723us) client_negotiation.cc:240] Negotiation successful
0924 16:03:42.576333 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3894,"mutex_wait_us":1,"thread_start_us":141,"threads_started":1}
I20260924 16:03:42.577669 5074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b83449b4193d4529bf4aa81d617a7f8f" candidate_uuid: "9ed3085a243e4385bf7113043238a389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ab8c81de7b44f32b8d8eb932872e58f" is_pre_election: true
I20260924 16:03:42.578013 5074 raft_consensus.cc:2454] T b83449b4193d4529bf4aa81d617a7f8f P 0ab8c81de7b44f32b8d8eb932872e58f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ed3085a243e4385bf7113043238a389 in term 0.
I20260924 16:03:42.578505 5075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b83449b4193d4529bf4aa81d617a7f8f" candidate_uuid: "9ed3085a243e4385bf7113043238a389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ab8c81de7b44f32b8d8eb932872e58f"
I20260924 16:03:42.578644 5075 raft_consensus.cc:3037] T b83449b4193d4529bf4aa81d617a7f8f P 0ab8c81de7b44f32b8d8eb932872e58f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:03:42.580206 5075 raft_consensus.cc:2454] T b83449b4193d4529bf4aa81d617a7f8f P 0ab8c81de7b44f32b8d8eb932872e58f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ed3085a243e4385bf7113043238a389 in term 1.
I20260924 16:03:42.603737 5431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:42.599263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56280 (local address 127.25.109.2:33377)
0924 16:03:42.599386 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 16:03:42.599393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:03:42.599409 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:03:42.599625 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:03:42.599628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:03:42.599705 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:03:42.599825 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:03:42.600089 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.600864 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.601875 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:03:42.602178 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:03:42.602277 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:03:42.602710 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:03:42.603305 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:03:42.603442 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 16:03:42.603515 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 16:03:42.603576 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 16:03:42.612159 5349 raft_consensus.cc:1261] T b83449b4193d4529bf4aa81d617a7f8f P 08025e5eca1045a09f304b7a33df9364 [term 1 FOLLOWER]: Refusing update from remote peer 9ed3085a243e4385bf7113043238a389: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:03:42.613059 5443 consensus_queue.cc:1035] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08025e5eca1045a09f304b7a33df9364" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 44603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:03:42.614070 5075 raft_consensus.cc:1261] T b83449b4193d4529bf4aa81d617a7f8f P 0ab8c81de7b44f32b8d8eb932872e58f [term 1 FOLLOWER]: Refusing update from remote peer 9ed3085a243e4385bf7113043238a389: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:03:42.614953 5443 consensus_queue.cc:1035] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ab8c81de7b44f32b8d8eb932872e58f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 32807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:03:42.627341 5450 mvcc.cc:205] Tried to move back new op lower bound from 7332928809410101248 to 7332928809245569024. Current Snapshot: MvccSnapshot[applied={T|T < 7332928809410101248}]
I20260924 16:03:42.633131 5447 mvcc.cc:205] Tried to move back new op lower bound from 7332928809410101248 to 7332928809245569024. Current Snapshot: MvccSnapshot[applied={T|T < 7332928809410101248 or (T in {7332928809410101248})}]
I20260924 16:03:42.636077 4925 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:03:42.636273 4925 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:03:42.639213 4925 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32f354e4573a4d768fa2110829981ba2: Sending DeleteTablet for 3 replicas of tablet b83449b4193d4529bf4aa81d617a7f8f
I20260924 16:03:42.639989 5347 tablet_service.cc:1514] Processing DeleteTablet for tablet b83449b4193d4529bf4aa81d617a7f8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:03:42 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:45282
I20260924 16:03:42.640295 5070 tablet_service.cc:1514] Processing DeleteTablet for tablet b83449b4193d4529bf4aa81d617a7f8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:03:42 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:45846
I20260924 16:03:42.641237 5212 tablet_service.cc:1514] Processing DeleteTablet for tablet b83449b4193d4529bf4aa81d617a7f8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:03:42 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:56264
I20260924 16:03:42.641804 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 5007
I20260924 16:03:42.641892 5459 tablet_replica.cc:330] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389: stopping tablet replica
I20260924 16:03:42.643213 5459 raft_consensus.cc:2229] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:03:42.643642 5459 raft_consensus.cc:2258] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:03:42.645220 5459 ts_tablet_manager.cc:1905] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:03:42.648237 5459 ts_tablet_manager.cc:1918] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:03:42.648348 5459 log.cc:1196] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-1/wal/wals/b83449b4193d4529bf4aa81d617a7f8f
I20260924 16:03:42.648723 5459 ts_tablet_manager.cc:1939] T b83449b4193d4529bf4aa81d617a7f8f P 9ed3085a243e4385bf7113043238a389: Deleting consensus metadata
I20260924 16:03:42.650032 4911 catalog_manager.cc:4841] TS 9ed3085a243e4385bf7113043238a389 (127.25.109.2:33377): tablet b83449b4193d4529bf4aa81d617a7f8f (table test-table [id=dcb3622ae5e1457da1b83d83bf2978de]) successfully deleted
I20260924 16:03:42.653409 5457 tablet_replica.cc:330] T b83449b4193d4529bf4aa81d617a7f8f P 08025e5eca1045a09f304b7a33df9364: stopping tablet replica
I20260924 16:03:42.653815 5457 raft_consensus.cc:2229] T b83449b4193d4529bf4aa81d617a7f8f P 08025e5eca1045a09f304b7a33df9364 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:03:42.654181 5457 raft_consensus.cc:2258] T b83449b4193d4529bf4aa81d617a7f8f P 08025e5eca1045a09f304b7a33df9364 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 16:03:42.658748 4913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.109.1:32807 (error 108)
I20260924 16:03:42.659714 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 5147
I20260924 16:03:42.661121 5457 ts_tablet_manager.cc:1905] T b83449b4193d4529bf4aa81d617a7f8f P 08025e5eca1045a09f304b7a33df9364: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:03:42.662526 5429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:03:42.662148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:32807 (local address 127.0.0.1:45848)
0924 16:03:42.662345 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 16:03:42.662429 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.1:32807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260924 16:03:42.662801 4913 catalog_manager.cc:4584] TS 0ab8c81de7b44f32b8d8eb932872e58f (127.25.109.1:32807): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b83449b4193d4529bf4aa81d617a7f8f: Network error: Client connection negotiation failed: client connection to 127.25.109.1:32807: connect: Connection refused (error 111)
I20260924 16:03:42.668870 5457 ts_tablet_manager.cc:1918] T b83449b4193d4529bf4aa81d617a7f8f P 08025e5eca1045a09f304b7a33df9364: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:03:42.669085 5457 log.cc:1196] T b83449b4193d4529bf4aa81d617a7f8f P 08025e5eca1045a09f304b7a33df9364: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265708917108-26036-0/minicluster-data/ts-2/wal/wals/b83449b4193d4529bf4aa81d617a7f8f
I20260924 16:03:42.669553 5457 ts_tablet_manager.cc:1939] T b83449b4193d4529bf4aa81d617a7f8f P 08025e5eca1045a09f304b7a33df9364: Deleting consensus metadata
I20260924 16:03:42.671290 4912 catalog_manager.cc:4841] TS 08025e5eca1045a09f304b7a33df9364 (127.25.109.3:44603): tablet b83449b4193d4529bf4aa81d617a7f8f (table test-table [id=dcb3622ae5e1457da1b83d83bf2978de]) successfully deleted
I20260924 16:03:42.671792 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 5284
I20260924 16:03:42.688985 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 4893
I20260924 16:03:42.697506 26036 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 16:03:43.042685 26036 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 16:03:43.042858 26036 mini_ranger.cc:66] Stopping Ranger...
I20260924 16:03:43.388384 26036 mini_ranger.cc:68] Stopped Ranger
2026-09-24 16:03:43.388 UTC [4476] LOG: received smart shutdown request
2026-09-24 16:03:43.392 UTC [4476] LOG: background worker "logical replication launcher" (PID 4485) exited with exit code 1
2026-09-24 16:03:43.392 UTC [4480] LOG: shutting down
2026-09-24 16:03:43.583 UTC [4476] LOG: database system is shut down
2026-09-24T16:03:43Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49772 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:03:43 dist-test-slave-n106 krb5kdc[5469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:03:43 dist-test-slave-n106 krb5kdc[5469](info): set up 2 sockets
Sep 24 16:03:43 dist-test-slave-n106 krb5kdc[5469](info): commencing operation
krb5kdc: starting...
W20260924 16:03:44.757614 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:03:44 dist-test-slave-n106 krb5kdc[5469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:03:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:03:44Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.25.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.25.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.25.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/rangeradmin_127.25.109.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.25.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/rangeradmin_127.25.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.25.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.25.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.25.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/rangerlookup_127.25.109.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.25.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/rangerlookup_127.25.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.25.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/HTTP_127.25.109.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.25.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/HTTP_127.25.109.20@KRBTEST.COM.keytab.
I20260924 16:03:44.927249 26036 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskCcKW0d/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/postgres -l logfile start
2026-09-24 16:03:47.423 UTC [5500] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 16:03:47.423 UTC [5500] LOG: listening on IPv4 address "127.25.109.20", port 34027
2026-09-24 16:03:47.427 UTC [5500] LOG: listening on Unix socket "/tmp/.s.PGSQL.34027"
2026-09-24 16:03:47.448 UTC [5503] LOG: database system was shut down at 2026-09-24 16:03:45 UTC
2026-09-24 16:03:47.452 UTC [5500] LOG: database system is ready to accept connections
I20260924 16:03:48.413419 26036 mini_postgres.cc:96] Postgres bound to 34027
2026-09-24 16:03:48.419 UTC [5512] FATAL: database "slave" does not exist
127.25.109.20:34027 - accepting connections
I20260924 16:03:48.420271 26036 mini_ranger.cc:161] Starting Ranger...
I20260924 16:03:48.438189 26036 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 16:03:48.924937 26036 mini_ranger.cc:87] Created ranger Postgres database
I20260924 16:03:48.925022 26036 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ranger-admin
2026-09-24 16:03:49,212 [I] DB FLAVOR :POSTGRES
2026-09-24 16:03:49,212 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 16:03:49,212 [I] Checking connection..
2026-09-24 16:03:49,213 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:34027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 16:03:49,476 [I] Checking connection passed.
2026-09-24 16:03:49,476 [I] --------- Verifying version history table ---------
2026-09-24 16:03:49,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:34027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 16:03:49,740 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 16:03:49,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:34027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 16:03:50,003 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 16:03:50,003 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 16:03:50,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:34027/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 16:03:50.260 UTC [5578] WARNING: there is no transaction in progress
2026-09-24 16:03:50,266 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 16:03:50,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:34027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 16:03:50,521 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 16:03:50,521 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 16:03:50,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:34027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 16:03:50,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:34027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 16:03:51,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:34027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-n106.c.gcp-upstream.internal','N') ;"
2026-09-24 16:03:51,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:34027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 16:03:51,496 [I] Table x_portal_user does not exist in database ranger
2026-09-24 16:03:51,497 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:34027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 16:03:51,750 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 16:03:51,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:34027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 16:03:52,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:34027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 16:03:52,228 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 16:03:52,228 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:34027/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 16:03:53.186 UTC [5688] WARNING: there is no transaction in progress
2026-09-24 16:03:53.203 UTC [5688] WARNING: there is no transaction in progress
2026-09-24 16:03:53.219 UTC [5688] WARNING: there is no transaction in progress
2026-09-24 16:03:53.235 UTC [5688] WARNING: there is no transaction in progress
2026-09-24 16:03:53.256 UTC [5688] WARNING: there is no transaction in progress
2026-09-24 16:03:53.274 UTC [5688] WARNING: there is no transaction in progress
2026-09-24 16:03:53.334 UTC [5688] WARNING: there is no transaction in progress
2026-09-24 16:03:53.341 UTC [5688] WARNING: there is no transaction in progress
2026-09-24 16:03:53.355 UTC [5688] WARNING: there is no transaction in progress
2026-09-24 16:03:53.365 UTC [5688] WARNING: there is no transaction in progress
2026-09-24 16:03:53.375 UTC [5688] WARNING: there is no transaction in progress
2026-09-24 16:03:53.388 UTC [5688] WARNING: there is no transaction in progress
2026-09-24 16:03:53.398 UTC [5688] WARNING: there is no transaction in progress
2026-09-24 16:03:53.404 UTC [5688] WARNING: there is no transaction in progress
2026-09-24 16:03:53.411 UTC [5688] WARNING: there is no transaction in progress
2026-09-24 16:03:53.417 UTC [5688] WARNING: there is no transaction in progress
2026-09-24 16:03:54,039 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 16:03:54,040 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:34027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 16:03:54,269 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 16:03:54,270 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:34027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 16:03:54,533 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 16:03:54,534 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:34027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 16:03:54,789 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 16:03:54,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:34027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 16:03:55,053 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:34027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 16:03:55,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:34027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-n106.c.gcp-upstream.internal';"
2026-09-24 16:03:55,589 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 16:03:55,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:34027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 16:03:55,841 [I] DB_PATCHES have already been applied
I20260924 16:03:55.846746 26036 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ranger-admin:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskCcKW0d/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 16:03:55.846812 26036 mini_ranger.cc:193] Using host: 127.25.109.20
I20260924 16:03:55.850394 26036 mini_ranger.cc:227] Ranger admin URL: http://127.25.109.20:37587
Sep 24, 2026 4:03:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:03:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:03:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 4:03:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 4:03:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 4:03:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 4:03:56 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 16:03:56.946218 26036 mini_ranger.cc:160] Time spent starting Ranger: real 8.526s user 0.003s sys 0.008s
Sep 24, 2026 4:04:16 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 4:04:16 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 4:04:16 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 4:04:16 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 4:04:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 4:04:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 4:04:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 4:04:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 4:04:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 4:04:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 4:04:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 4:04:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 4:04:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 4:04:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 4:04:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 4:04:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 4:04:16 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 4:04:17 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 16:04:17.926020 26036 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.25.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.25.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.25.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/rangerkms_127.25.109.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.25.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/rangerkms_127.25.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.25.109.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.109.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.109.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/HTTP_127.25.109.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.25.109.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/HTTP_127.25.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 16:04:18 dist-test-slave-n106 krb5kdc[5469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265858, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 16:04:18.013633 26036 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 16:04:18.023238 26036 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 16:04:18.875492 26036 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 16:04:18.875586 26036 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ranger-kms
I20260924 16:04:18.875608 26036 mini_ranger_kms.cc:227] Using postgres at 127.25.109.20:34027
2026-09-24 16:04:19,089 [I] DB FLAVOR :POSTGRES
2026-09-24 16:04:19,089 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 16:04:19,089 [I] Checking connection
2026-09-24 16:04:19,089 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:34027/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 16:04:19,373 [I] connection success
2026-09-24 16:04:19,374 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 16:04:19,374 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 16:04:19,374 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:34027/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 16:04:19,609 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 16:04:19,610 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 16:04:19,610 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 16:04:19,610 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.109.20:34027/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 16:04:19,862 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 16:04:20.347038 26036 mini_ranger_kms.cc:310] Created kms service
I20260924 16:04:20.545961 26036 mini_ranger_kms.cc:326] Created kudu user
I20260924 16:04:20.588449 26036 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 16:04:20 dist-test-slave-n106 krb5kdc[5469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265858, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.25.109.20@KRBTEST.COM
I20260924 16:04:20.995366 26036 mini_ranger_kms.cc:384] Added ranger policy
I20260924 16:04:20.995518 26036 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ranger-kms:/tmp/dist-test-taskCcKW0d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskCcKW0d/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 16:04:20.995540 26036 mini_ranger_kms.cc:240] Using host: 127.25.109.20
I20260924 16:04:20.999356 26036 mini_ranger_kms.cc:276] Ranger KMS PID: 5870
I20260924 16:04:20.999436 26036 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.25.109.20:37853
Sep 24, 2026 4:04:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:04:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:04:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 4:04:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 4:04:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 4:04:22 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 24 16:04:23 dist-test-slave-n106 krb5kdc[5469](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790265863, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.109.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:23.239954 26036 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.226s user 0.000s sys 0.196s
I20260924 16:04:23.240095 26036 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 16:04:23.240176 26036 mini_ranger_kms.cc:401] 127.25.109.20:37853/kms/v1/keys
Sep 24 16:04:23 dist-test-slave-n106 krb5kdc[5469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265863, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.109.20@KRBTEST.COM for HTTP/127.25.109.20@KRBTEST.COM
Sep 24 16:04:24 dist-test-slave-n106 krb5kdc[5469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265863, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.109.20@KRBTEST.COM for HTTP/127.25.109.20@KRBTEST.COM
Sep 24 16:04:26 dist-test-slave-n106 krb5kdc[5469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
I20260924 16:04:26.505530 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:32817
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:32817
--ranger_config_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.109.20:37853/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:26.618257 5923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:26.618507 5923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:26.618548 5923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:26.622562 5923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:04:26.622620 5923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:26.622644 5923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:26.622660 5923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:04:26.622675 5923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:04:26.628121 5923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/wal
--ranger_kms_url=127.25.109.20:37853/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:32817
--ranger_config_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:32817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:04:26.629235 5923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:26.630330 5923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:26.641613 5928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:26.642066 5923 server_base.cc:1034] running on GCE node
W20260924 16:04:26.641630 5929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:26.641809 5931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:26.642956 5923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:26.644166 5923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:26.645344 5923 hybrid_clock.cc:648] HybridClock initialized: now 1790265866645320 us; error 45 us; skew 500 ppm
Sep 24 16:04:26 dist-test-slave-n106 krb5kdc[5469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265866, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:26.652985 5923 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:04:26.653926 5923 webserver.cc:460] Webserver started at http://127.25.109.62:35521/ using document root <none> and password file <none>
I20260924 16:04:26.654470 5923 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:26.654524 5923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:26.654757 5923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:04:26 dist-test-slave-n106 krb5kdc[5469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265866, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for HTTP/127.25.109.20@KRBTEST.COM
I20260924 16:04:26.712044 5923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "931a69552f774dd18e9a7ac45276270d"
format_stamp: "Formatted at 2026-09-24 16:04:26 on dist-test-slave-n106"
server_key: "d21e5d3098a4895c73f7227e10bbebc3"
server_key_iv: "2d9b07d009a50de5c3aab12fc9d07260"
server_key_version: "kuduclusterkey@0"
I20260924 16:04:26.712860 5923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "931a69552f774dd18e9a7ac45276270d"
format_stamp: "Formatted at 2026-09-24 16:04:26 on dist-test-slave-n106"
server_key: "d21e5d3098a4895c73f7227e10bbebc3"
server_key_iv: "2d9b07d009a50de5c3aab12fc9d07260"
server_key_version: "kuduclusterkey@0"
I20260924 16:04:26.717945 5923 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 16:04:26.736907 5938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:26.738632 5923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:04:26.738781 5923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "931a69552f774dd18e9a7ac45276270d"
format_stamp: "Formatted at 2026-09-24 16:04:26 on dist-test-slave-n106"
server_key: "d21e5d3098a4895c73f7227e10bbebc3"
server_key_iv: "2d9b07d009a50de5c3aab12fc9d07260"
server_key_version: "kuduclusterkey@0"
I20260924 16:04:26.738926 5923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:26.760005 5923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:26.763492 5923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:26.763695 5923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:26.772627 5923 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:32817
I20260924 16:04:26.772637 5990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:32817 every 8 connection(s)
I20260924 16:04:26.773874 5923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:04:26.776809 5991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:26.782239 5991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d: Bootstrap starting.
I20260924 16:04:26.783315 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 5923
I20260924 16:04:26.783478 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:04:26.783799 26036 external_mini_cluster.cc:1428] Setting key f834771ab28ea37659dd08543a91c1e9
I20260924 16:04:26.785338 5991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:26.786079 5991 log.cc:824] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:26.788271 5991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d: No bootstrap required, opened a new log
Sep 24 16:04:26 dist-test-slave-n106 krb5kdc[5469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:04:26.792824 5991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931a69552f774dd18e9a7ac45276270d" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32817 } }
I20260924 16:04:26.793124 5991 raft_consensus.cc:374] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:26.793206 5991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 931a69552f774dd18e9a7ac45276270d, State: Initialized, Role: FOLLOWER
I20260924 16:04:26.793700 5991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931a69552f774dd18e9a7ac45276270d" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32817 } }
I20260924 16:04:26.793887 5991 raft_consensus.cc:388] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:04:26.793962 5991 raft_consensus.cc:482] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:04:26.794065 5991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:26.794916 5991 raft_consensus.cc:504] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931a69552f774dd18e9a7ac45276270d" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32817 } }
I20260924 16:04:26.795261 5991 leader_election.cc:302] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 931a69552f774dd18e9a7ac45276270d; no voters:
I20260924 16:04:26.795608 5991 leader_election.cc:288] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:04:26.795704 5996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:26.795907 5996 raft_consensus.cc:686] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [term 1 LEADER]: Becoming Leader. State: Replica: 931a69552f774dd18e9a7ac45276270d, State: Running, Role: LEADER
I20260924 16:04:26.796257 5996 consensus_queue.cc:237] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931a69552f774dd18e9a7ac45276270d" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32817 } }
I20260924 16:04:26.797016 5991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:04:26.798277 5997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "931a69552f774dd18e9a7ac45276270d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931a69552f774dd18e9a7ac45276270d" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32817 } } }
I20260924 16:04:26.798318 5998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 931a69552f774dd18e9a7ac45276270d. Latest consensus state: current_term: 1 leader_uuid: "931a69552f774dd18e9a7ac45276270d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931a69552f774dd18e9a7ac45276270d" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32817 } } }
I20260924 16:04:26.798557 5997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:26.798588 5998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:26.798738 5994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:26.785025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52216 (local address 127.25.109.62:32817)
0924 16:04:26.785449 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:26.785471 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:26.785516 (+ 45us) server_negotiation.cc:407] Connection header received
0924 16:04:26.786144 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:26.786160 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:26.786612 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:26.786973 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:26.787833 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:26.788896 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:26.790152 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:26.790617 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:26.790734 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:26.793569 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:26.793575 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:26.793590 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:26.793621 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:26.795425 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:26.795960 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:26.795967 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:26.795973 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:26.796097 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:26.796380 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:26.796385 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:26.796386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:26.796834 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:26.797044 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:26.797351 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 16:04:26.797667 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":137,"threads_started":1}
I20260924 16:04:26.799758 5991 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 16:04:26.803659 6006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:04:26.807749 6006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:04:26.813987 6006 catalog_manager.cc:1347] Generated new cluster ID: 1aea77f81ba741a2b2e2ce6b1c1094af
I20260924 16:04:26.814055 6006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:04:26.831931 6006 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:04:26.833278 6006 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:04:26.845270 6006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d: Generated new TSK 0
I20260924 16:04:26.846166 6006 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
I20260924 16:04:27.535844 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.109.20:37853/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:0
--local_ip_for_outbound_sockets=127.25.109.1
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:32817
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:27.679054 6037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:27.679327 6037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:27.679432 6037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:27.684592 6037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:27.684659 6037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:27.684742 6037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:04:27.691181 6037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.25.109.20:37853/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:32817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:04:27.692452 6037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:27.693631 6037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:27.705052 6044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:27.705202 6037 server_base.cc:1034] running on GCE node
W20260924 16:04:27.705082 6045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:27.708567 6047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:27.709023 6037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:27.709800 6037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:27.710980 6037 hybrid_clock.cc:648] HybridClock initialized: now 1790265867710945 us; error 46 us; skew 500 ppm
Sep 24 16:04:27 dist-test-slave-n106 krb5kdc[5469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265867, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:27.713832 6037 init.cc:377] Logged in from keytab as kudu/127.25.109.1@KRBTEST.COM (short username kudu)
I20260924 16:04:27.714722 6037 webserver.cc:460] Webserver started at http://127.25.109.1:35801/ using document root <none> and password file <none>
I20260924 16:04:27.715407 6037 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:27.715509 6037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:27.715817 6037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:04:27 dist-test-slave-n106 krb5kdc[5469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265867, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for HTTP/127.25.109.20@KRBTEST.COM
I20260924 16:04:27.732148 6037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-0/data/instance:
uuid: "5df65559974a4de0b981bc7581fcbe79"
format_stamp: "Formatted at 2026-09-24 16:04:27 on dist-test-slave-n106"
server_key: "824971c7680b02f4e28a3338325f022d"
server_key_iv: "20b22d055e70db334d1a1e799a245320"
server_key_version: "kuduclusterkey@0"
I20260924 16:04:27.732971 6037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance:
uuid: "5df65559974a4de0b981bc7581fcbe79"
format_stamp: "Formatted at 2026-09-24 16:04:27 on dist-test-slave-n106"
server_key: "824971c7680b02f4e28a3338325f022d"
server_key_iv: "20b22d055e70db334d1a1e799a245320"
server_key_version: "kuduclusterkey@0"
I20260924 16:04:27.737022 6037 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:04:27.754292 6054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:27.756000 6037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:04:27.756127 6037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "5df65559974a4de0b981bc7581fcbe79"
format_stamp: "Formatted at 2026-09-24 16:04:27 on dist-test-slave-n106"
server_key: "824971c7680b02f4e28a3338325f022d"
server_key_iv: "20b22d055e70db334d1a1e799a245320"
server_key_version: "kuduclusterkey@0"
I20260924 16:04:27.756253 6037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:27.796315 6037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:27.799962 6037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:27.800196 6037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:27.800923 6037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:27.801961 6037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:27.802054 6037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:27.802141 6037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:27.802196 6037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:27.816248 6037 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:37067
I20260924 16:04:27.816535 6171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:37067 every 8 connection(s)
I20260924 16:04:27.817667 6037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:04:27.825559 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 6037
I20260924 16:04:27.825706 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance
I20260924 16:04:27.826047 26036 external_mini_cluster.cc:1428] Setting key a8635bed422128dec8a0191218752807
Sep 24 16:04:27 dist-test-slave-n106 krb5kdc[5469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265867, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:04:27.835166 6174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:27.819849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:45693 (local address 127.25.109.62:32817)
0924 16:04:27.820153 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:27.820161 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:27.821232 (+ 1071us) server_negotiation.cc:407] Connection header received
0924 16:04:27.822477 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:27.822493 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:27.822551 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:27.822613 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:27.824227 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.825082 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.826818 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.827182 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.827257 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:27.830712 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:27.830716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:27.830719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:27.830745 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:27.832708 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:27.833339 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:27.833343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:27.833345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:27.833437 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:27.833843 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:27.833846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:27.833848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:27.834072 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:27.834143 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:27.834846 (+ 703us) server_negotiation.cc:299] Negotiation successful
0924 16:04:27.834979 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":108,"threads_started":1}
I20260924 16:04:27.836869 6175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:27.820304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:32817 (local address 127.25.109.1:45693)
0924 16:04:27.821064 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:27.821122 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:27.822222 (+ 1100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:27.823019 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:27.823027 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:27.823363 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:27.824041 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:27.824061 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.825233 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.825237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:27.826672 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:27.826684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:27.827965 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:27.827969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:27.828173 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:27.828845 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:27.828865 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:27.830524 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:27.832853 (+ 2329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:27.832862 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:27.832880 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:27.833217 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:27.833561 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:27.833564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:27.833566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:27.833738 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:27.834224 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:27.834231 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:27.834561 (+ 330us) client_negotiation.cc:769] Sending connection context
0924 16:04:27.835398 (+ 837us) client_negotiation.cc:240] Negotiation successful
0924 16:04:27.835761 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":125,"threads_started":1}
I20260924 16:04:27.838644 6172 heartbeater.cc:347] Connected to a master server at 127.25.109.62:32817
I20260924 16:04:27.838896 6172 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.25.109.2@KRBTEST.COM; defaulting to no policy
I20260924 16:04:27.839501 6172 heartbeater.cc:511] Master 127.25.109.62:32817 requested a full tablet report, sending...
I20260924 16:04:27.841287 5955 ts_manager.cc:194] Registered new tserver with Master: 5df65559974a4de0b981bc7581fcbe79 (127.25.109.1:37067)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.2@KRBTEST.COM" created.
I20260924 16:04:27.842854 5955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.1@KRBTEST.COM'} at 127.25.109.1:45693
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
I20260924 16:04:27.860385 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.109.20:37853/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:0
--local_ip_for_outbound_sockets=127.25.109.2
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:32817
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:04:27 dist-test-slave-n106 krb5kdc[5469](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790265867, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 16:04:28.026441 6178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:28.026832 6178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:28.026903 6178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:28.033011 6178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:28.033138 6178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:28.033250 6178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:04:28.041365 6178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.25.109.20:37853/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:32817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:04:28.043020 6178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:28.044574 6178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:28.057467 6184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:28.057525 6185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:28.057723 6187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:28.058530 6178 server_base.cc:1034] running on GCE node
I20260924 16:04:28.059074 6178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:28.059976 6178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:28.061170 6178 hybrid_clock.cc:648] HybridClock initialized: now 1790265868061095 us; error 89 us; skew 500 ppm
Sep 24 16:04:28 dist-test-slave-n106 krb5kdc[5469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265868, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:28.064961 6178 init.cc:377] Logged in from keytab as kudu/127.25.109.2@KRBTEST.COM (short username kudu)
I20260924 16:04:28.066020 6178 webserver.cc:460] Webserver started at http://127.25.109.2:45997/ using document root <none> and password file <none>
I20260924 16:04:28.066779 6178 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:28.066860 6178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:28.067075 6178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:04:28 dist-test-slave-n106 krb5kdc[5469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265868, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for HTTP/127.25.109.20@KRBTEST.COM
I20260924 16:04:28.083484 6178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/data/instance:
uuid: "f179660025c54890ab5560d79c492041"
format_stamp: "Formatted at 2026-09-24 16:04:28 on dist-test-slave-n106"
server_key: "6c8bd8e884d180093796f395f08b23f1"
server_key_iv: "8353b35bb08e1e113ad652383b00bf4a"
server_key_version: "kuduclusterkey@0"
I20260924 16:04:28.084301 6178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance:
uuid: "f179660025c54890ab5560d79c492041"
format_stamp: "Formatted at 2026-09-24 16:04:28 on dist-test-slave-n106"
server_key: "6c8bd8e884d180093796f395f08b23f1"
server_key_iv: "8353b35bb08e1e113ad652383b00bf4a"
server_key_version: "kuduclusterkey@0"
I20260924 16:04:28.089751 6178 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260924 16:04:28.108754 6195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:28.110548 6178 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:04:28.110675 6178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "f179660025c54890ab5560d79c492041"
format_stamp: "Formatted at 2026-09-24 16:04:28 on dist-test-slave-n106"
server_key: "6c8bd8e884d180093796f395f08b23f1"
server_key_iv: "8353b35bb08e1e113ad652383b00bf4a"
server_key_version: "kuduclusterkey@0"
I20260924 16:04:28.110812 6178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:28.141239 6178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:28.148131 6178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:28.148478 6178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:28.149267 6178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:28.150393 6178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:28.150508 6178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:28.150614 6178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:28.150677 6178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:28.164670 6178 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:39517
I20260924 16:04:28.164749 6309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:39517 every 8 connection(s)
I20260924 16:04:28.165879 6178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:04:28.168654 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 6178
I20260924 16:04:28.168785 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance
I20260924 16:04:28.169149 26036 external_mini_cluster.cc:1428] Setting key 46a1f2c2aefbaa231dbcd9bfdaa109db
Sep 24 16:04:28 dist-test-slave-n106 krb5kdc[5469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265868, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.109.3@KRBTEST.COM; defaulting to no policy
I20260924 16:04:28.186350 6174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:28.168040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:52583 (local address 127.25.109.62:32817)
0924 16:04:28.168203 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:28.168207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:28.172985 (+ 4778us) server_negotiation.cc:407] Connection header received
0924 16:04:28.174105 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:28.174109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:28.174167 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:28.174248 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:28.175394 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:28.175939 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:28.177839 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:28.178281 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:28.178368 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:28.181743 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:28.181749 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:28.181753 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:28.181785 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:28.183778 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:28.184481 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:28.184485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:28.184487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:28.184599 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:28.185063 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:28.185067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:28.185068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:28.185269 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:28.185384 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:28.185999 (+ 615us) server_negotiation.cc:299] Negotiation successful
0924 16:04:28.186154 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.3@KRBTEST.COM" created.
I20260924 16:04:28.187459 6312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:28.168644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:32817 (local address 127.25.109.2:52583)
0924 16:04:28.172801 (+ 4157us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:28.172875 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:28.173876 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:28.174435 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:28.174442 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:28.174733 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:28.175239 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:28.175252 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:28.176082 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:28.176088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:28.177670 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:28.177687 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:28.178638 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:28.178642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:28.178796 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:28.179474 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:28.179501 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:28.181504 (+ 2003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:28.183934 (+ 2430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:28.183943 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:28.183964 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:28.184321 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:28.184727 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:28.184733 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:28.184737 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:28.184934 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:28.185384 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:28.185393 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:28.185744 (+ 351us) client_negotiation.cc:769] Sending connection context
0924 16:04:28.185944 (+ 200us) client_negotiation.cc:240] Negotiation successful
0924 16:04:28.186286 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3866,"thread_start_us":129,"threads_started":1}
I20260924 16:04:28.189394 6310 heartbeater.cc:347] Connected to a master server at 127.25.109.62:32817
I20260924 16:04:28.189682 6310 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:28.190327 6310 heartbeater.cc:511] Master 127.25.109.62:32817 requested a full tablet report, sending...
I20260924 16:04:28.191495 5955 ts_manager.cc:194] Registered new tserver with Master: f179660025c54890ab5560d79c492041 (127.25.109.2:39517)
I20260924 16:04:28.192101 5955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.2@KRBTEST.COM'} at 127.25.109.2:52583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
I20260924 16:04:28.228179 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.109.20:37853/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:0
--local_ip_for_outbound_sockets=127.25.109.3
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:32817
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:28.411181 6315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:28.411777 6315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:28.411990 6315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:28.422837 6315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:28.423022 6315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:28.423162 6315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:04:28.432695 6315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.25.109.20:37853/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:32817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:04:28.434500 6315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
Sep 24 16:04:28 dist-test-slave-n106 krb5kdc[5469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265867, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for HTTP/127.25.109.20@KRBTEST.COM
I20260924 16:04:28.436118 6315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:28.453181 6320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:28.454969 6321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:28.461136 6323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:28.461647 6315 server_base.cc:1034] running on GCE node
I20260924 16:04:28.462898 6315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:28.463815 6315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:28.481416 6315 hybrid_clock.cc:648] HybridClock initialized: now 1790265868481302 us; error 139 us; skew 500 ppm
Sep 24 16:04:28 dist-test-slave-n106 krb5kdc[5469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265868, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:28.487656 6315 init.cc:377] Logged in from keytab as kudu/127.25.109.3@KRBTEST.COM (short username kudu)
I20260924 16:04:28.489151 6315 webserver.cc:460] Webserver started at http://127.25.109.3:36919/ using document root <none> and password file <none>
I20260924 16:04:28.490049 6315 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:28.490212 6315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:28.490541 6315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:04:28 dist-test-slave-n106 krb5kdc[5469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265868, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for HTTP/127.25.109.20@KRBTEST.COM
I20260924 16:04:28.512315 6315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/data/instance:
uuid: "a1d97c6104bb49248d80d5999a11375d"
format_stamp: "Formatted at 2026-09-24 16:04:28 on dist-test-slave-n106"
server_key: "df32f031628b0c516b4e23e2407cc2d1"
server_key_iv: "9d6ce8fcb4002811eb0390f6a64a12b9"
server_key_version: "kuduclusterkey@0"
I20260924 16:04:28.522802 6315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance:
uuid: "a1d97c6104bb49248d80d5999a11375d"
format_stamp: "Formatted at 2026-09-24 16:04:28 on dist-test-slave-n106"
server_key: "df32f031628b0c516b4e23e2407cc2d1"
server_key_iv: "9d6ce8fcb4002811eb0390f6a64a12b9"
server_key_version: "kuduclusterkey@0"
I20260924 16:04:28.537477 6315 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.004s sys 0.000s
I20260924 16:04:28.559741 6331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:28.561616 6315 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:04:28.561743 6315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "a1d97c6104bb49248d80d5999a11375d"
format_stamp: "Formatted at 2026-09-24 16:04:28 on dist-test-slave-n106"
server_key: "df32f031628b0c516b4e23e2407cc2d1"
server_key_iv: "9d6ce8fcb4002811eb0390f6a64a12b9"
server_key_version: "kuduclusterkey@0"
I20260924 16:04:28.561872 6315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:28.595880 6315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:28.599642 6315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:28.599968 6315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:28.600768 6315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:28.601847 6315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:28.601950 6315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:28.602051 6315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:28.602118 6315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:28.615825 6315 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:42129
I20260924 16:04:28.617101 6315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:04:28.620499 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 6315
I20260924 16:04:28.620622 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance
I20260924 16:04:28.620919 26036 external_mini_cluster.cc:1428] Setting key f518da1b48a1267b416409c86a56e8fb
I20260924 16:04:28.625494 6447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:42129 every 8 connection(s)
Sep 24 16:04:28 dist-test-slave-n106 krb5kdc[5469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265868, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:04:28.645099 6174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:28.629265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:58803 (local address 127.25.109.62:32817)
0924 16:04:28.629381 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:28.629385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:28.630126 (+ 741us) server_negotiation.cc:407] Connection header received
0924 16:04:28.631373 (+ 1247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:28.631377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:28.631431 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:28.631482 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:28.633090 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:28.633798 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:28.637047 (+ 3249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:28.637363 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:28.637442 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:28.640587 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:28.640591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:28.640594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:28.640624 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:28.642390 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:28.643021 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:28.643024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:28.643026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:28.643115 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:28.643592 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:28.643596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:28.643597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:28.643820 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:28.643905 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:28.644800 (+ 895us) server_negotiation.cc:299] Negotiation successful
0924 16:04:28.644921 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 16:04:28.648784 6450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:28.628189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:32817 (local address 127.25.109.3:58803)
0924 16:04:28.629984 (+ 1795us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:28.630039 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:28.631097 (+ 1058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:28.631883 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:28.631892 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:28.632242 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:28.632919 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:28.632936 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:28.635485 (+ 2549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:28.635489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:28.636883 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:28.636895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:28.637651 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:28.637654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:28.637852 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:28.638592 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:28.638614 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:28.640376 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:28.642529 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:28.642535 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:28.642551 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:28.642887 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:28.643225 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:28.643229 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:28.643233 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:28.643438 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:28.643984 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:28.643990 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:28.644427 (+ 437us) client_negotiation.cc:769] Sending connection context
0924 16:04:28.647232 (+ 2805us) client_negotiation.cc:240] Negotiation successful
0924 16:04:28.647606 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1562,"thread_start_us":170,"threads_started":1}
I20260924 16:04:28.650828 6448 heartbeater.cc:347] Connected to a master server at 127.25.109.62:32817
I20260924 16:04:28.651084 6448 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:28.651651 6448 heartbeater.cc:511] Master 127.25.109.62:32817 requested a full tablet report, sending...
I20260924 16:04:28.652895 5955 ts_manager.cc:194] Registered new tserver with Master: a1d97c6104bb49248d80d5999a11375d (127.25.109.3:42129)
I20260924 16:04:28.653645 5955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.3@KRBTEST.COM'} at 127.25.109.3:58803
I20260924 16:04:28.658839 26036 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:04:28.659672 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 6037
I20260924 16:04:28.670337 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 6178
I20260924 16:04:28.677585 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 6315
I20260924 16:04:28.684648 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 5923
I20260924 16:04:28.692910 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:32817
--webserver_interface=127.25.109.62
--webserver_port=35521
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:32817
--ranger_config_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.109.20:37853/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.25.109.20:37853/kms with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:28.804330 6451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:28.804687 6451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:28.804726 6451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:28.808832 6451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:04:28.808894 6451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:28.808913 6451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:28.808929 6451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:04:28.808964 6451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:04:28.814404 6451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.25.109.20:37853/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:32817
--ranger_config_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:32817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=35521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:04:28.815462 6451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:28.816553 6451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:28.825001 6457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:28.825047 6456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:28.825176 6451 server_base.cc:1034] running on GCE node
W20260924 16:04:28.825340 6459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:28.825889 6451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:28.826956 6451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:28.828142 6451 hybrid_clock.cc:648] HybridClock initialized: now 1790265868828109 us; error 47 us; skew 500 ppm
Sep 24 16:04:28 dist-test-slave-n106 krb5kdc[5469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265868, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:28.830798 6451 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:04:28.831650 6451 webserver.cc:460] Webserver started at http://127.25.109.62:35521/ using document root <none> and password file <none>
I20260924 16:04:28.832199 6451 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:28.832293 6451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 16:04:28 dist-test-slave-n106 krb5kdc[5469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265868, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for HTTP/127.25.109.20@KRBTEST.COM
I20260924 16:04:28.893215 6451 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:04:28.896078 6467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:28.897480 6451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:04:28.897629 6451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "931a69552f774dd18e9a7ac45276270d"
format_stamp: "Formatted at 2026-09-24 16:04:26 on dist-test-slave-n106"
server_key: "d21e5d3098a4895c73f7227e10bbebc3"
server_key_iv: "2d9b07d009a50de5c3aab12fc9d07260"
server_key_version: "kuduclusterkey@0"
I20260924 16:04:28.898201 6451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:28.910368 6451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:28.913125 6451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:28.913331 6451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:28.920348 6451 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:32817
I20260924 16:04:28.920354 6521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:32817 every 8 connection(s)
I20260924 16:04:28.921452 6451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:04:28.925783 6522 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:04:28.927383 6522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d: Bootstrap starting.
I20260924 16:04:28.929922 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 6451
I20260924 16:04:28.930359 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.109.20:37853/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:37067
--local_ip_for_outbound_sockets=127.25.109.1
--tserver_master_addrs=127.25.109.62:32817
--webserver_port=35801
--webserver_interface=127.25.109.1
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.25.109.20:37853/kms with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:04:28.943900 6522 log.cc:824] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:28.950659 6522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:04:28.951061 6522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d: Bootstrap complete.
I20260924 16:04:28.955235 6522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931a69552f774dd18e9a7ac45276270d" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32817 } }
I20260924 16:04:28.955644 6522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 931a69552f774dd18e9a7ac45276270d, State: Initialized, Role: FOLLOWER
I20260924 16:04:28.956146 6522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931a69552f774dd18e9a7ac45276270d" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32817 } }
I20260924 16:04:28.956292 6522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:04:28.956382 6522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:04:28.956523 6522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [term 1 FOLLOWER]: Advancing to term 2
I20260924 16:04:28.958381 6522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931a69552f774dd18e9a7ac45276270d" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32817 } }
I20260924 16:04:28.958712 6522 leader_election.cc:302] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 931a69552f774dd18e9a7ac45276270d; no voters:
I20260924 16:04:28.959116 6522 leader_election.cc:288] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 16:04:28.959182 6526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [term 2 FOLLOWER]: Leader election won for term 2
I20260924 16:04:28.959367 6526 raft_consensus.cc:686] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [term 2 LEADER]: Becoming Leader. State: Replica: 931a69552f774dd18e9a7ac45276270d, State: Running, Role: LEADER
I20260924 16:04:28.959731 6526 consensus_queue.cc:237] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931a69552f774dd18e9a7ac45276270d" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32817 } }
I20260924 16:04:28.960325 6522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:04:28.961215 6528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "931a69552f774dd18e9a7ac45276270d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931a69552f774dd18e9a7ac45276270d" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32817 } } }
I20260924 16:04:28.961254 6527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 931a69552f774dd18e9a7ac45276270d. Latest consensus state: current_term: 2 leader_uuid: "931a69552f774dd18e9a7ac45276270d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931a69552f774dd18e9a7ac45276270d" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32817 } } }
I20260924 16:04:28.961388 6527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:28.961337 6528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:28.961772 6535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:04:28.965272 6535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:04:28.965824 6522 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 16:04:28.965970 6535 catalog_manager.cc:1259] Loaded cluster ID: 1aea77f81ba741a2b2e2ce6b1c1094af
I20260924 16:04:28.966018 6535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:04:28.968340 6535 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:04:28.969033 6535 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d: Loaded TSK: 0
I20260924 16:04:28.970238 6535 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 16:04:29.074709 6524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:29.075151 6524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:29.075287 6524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:29.081563 6524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:29.081713 6524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:29.081861 6524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:04:29.090024 6524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.25.109.20:37853/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:37067
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=35801
--tserver_master_addrs=127.25.109.62:32817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:04:29.091382 6524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:29.092761 6524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:29.105290 6561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:29.105284 6560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:29.105965 6524 server_base.cc:1034] running on GCE node
W20260924 16:04:29.106463 6563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:29.107026 6524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:29.107846 6524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:29.111150 6524 hybrid_clock.cc:648] HybridClock initialized: now 1790265869111106 us; error 49 us; skew 500 ppm
Sep 24 16:04:29 dist-test-slave-n106 krb5kdc[5469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265869, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:29.114768 6524 init.cc:377] Logged in from keytab as kudu/127.25.109.1@KRBTEST.COM (short username kudu)
I20260924 16:04:29.115803 6524 webserver.cc:460] Webserver started at http://127.25.109.1:35801/ using document root <none> and password file <none>
I20260924 16:04:29.116501 6524 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:29.116667 6524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 16:04:29 dist-test-slave-n106 krb5kdc[5469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265869, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for HTTP/127.25.109.20@KRBTEST.COM
I20260924 16:04:29.143949 6524 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.000s
I20260924 16:04:29.148720 6571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:29.150432 6524 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260924 16:04:29.150640 6524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "5df65559974a4de0b981bc7581fcbe79"
format_stamp: "Formatted at 2026-09-24 16:04:27 on dist-test-slave-n106"
server_key: "824971c7680b02f4e28a3338325f022d"
server_key_iv: "20b22d055e70db334d1a1e799a245320"
server_key_version: "kuduclusterkey@0"
I20260924 16:04:29.151479 6524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:29.177337 6524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:29.180629 6524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:29.180841 6524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:29.181523 6524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:29.182411 6524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:29.182478 6524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:29.182564 6524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:29.182619 6524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:29.193738 6524 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:37067
I20260924 16:04:29.193763 6687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:37067 every 8 connection(s)
I20260924 16:04:29.194829 6524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:04:29.197929 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 6524
I20260924 16:04:29.198419 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.109.20:37853/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:39517
--local_ip_for_outbound_sockets=127.25.109.2
--tserver_master_addrs=127.25.109.62:32817
--webserver_port=45997
--webserver_interface=127.25.109.2
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.25.109.20:37853/kms with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:04:29 dist-test-slave-n106 krb5kdc[5469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265869, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:04:29.219074 6690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:29.197260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:32817 (local address 127.25.109.1:45819)
0924 16:04:29.200286 (+ 3026us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:29.200341 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:29.201513 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:29.203356 (+ 1843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:29.203364 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:29.203729 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:29.204329 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:29.204347 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.206290 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.206293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:29.207134 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:29.207144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.208073 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.208078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:29.208265 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:29.208947 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:29.208967 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:29.212831 (+ 3864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:29.215457 (+ 2626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:29.215463 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:29.215478 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:29.215822 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:29.216223 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:29.216226 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:29.216228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:29.216390 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:29.217196 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:29.217202 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:29.217507 (+ 305us) client_negotiation.cc:769] Sending connection context
0924 16:04:29.217714 (+ 207us) client_negotiation.cc:240] Negotiation successful
0924 16:04:29.218034 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":133,"threads_started":1}
I20260924 16:04:29.221104 6692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:29.198483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:45819 (local address 127.25.109.62:32817)
0924 16:04:29.199107 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:29.199132 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:29.200499 (+ 1367us) server_negotiation.cc:407] Connection header received
0924 16:04:29.202108 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:29.202123 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:29.202533 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:29.202864 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:29.204552 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.206141 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.207269 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.207894 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.208078 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:29.213062 (+ 4984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:29.213069 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:29.213085 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:29.213119 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:29.215307 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:29.215963 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:29.215969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:29.215975 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:29.216114 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:29.216531 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:29.216534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:29.216536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:29.217006 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:29.217140 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:29.219875 (+ 2735us) server_negotiation.cc:299] Negotiation successful
0924 16:04:29.220168 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":112,"threads_started":1}
I20260924 16:04:29.224797 6688 heartbeater.cc:347] Connected to a master server at 127.25.109.62:32817
I20260924 16:04:29.225039 6688 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:29.225607 6688 heartbeater.cc:511] Master 127.25.109.62:32817 requested a full tablet report, sending...
W20260924 16:04:29.227866 6688 heartbeater.cc:644] Failed to heartbeat to 127.25.109.62:32817 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 16:04:29.227979 6688 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:29.228147 6688 heartbeater.cc:511] Master 127.25.109.62:32817 requested a full tablet report, sending...
I20260924 16:04:29.229144 6688 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:29.229305 6688 heartbeater.cc:511] Master 127.25.109.62:32817 requested a full tablet report, sending...
W20260924 16:04:29.230134 6688 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 16:04:29.374830 6693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:29.375170 6693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:29.375289 6693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:29.381067 6693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:29.381150 6693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:29.381250 6693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:04:29.386842 6693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.25.109.20:37853/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:39517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=45997
--tserver_master_addrs=127.25.109.62:32817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:04:29.387903 6693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:29.389025 6693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:29.401360 6699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:29.401677 6698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:29.402338 6701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:29.402853 6693 server_base.cc:1034] running on GCE node
I20260924 16:04:29.403364 6693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:29.404204 6693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:29.405386 6693 hybrid_clock.cc:648] HybridClock initialized: now 1790265869405352 us; error 47 us; skew 500 ppm
Sep 24 16:04:29 dist-test-slave-n106 krb5kdc[5469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265869, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:29.409029 6693 init.cc:377] Logged in from keytab as kudu/127.25.109.2@KRBTEST.COM (short username kudu)
I20260924 16:04:29.410132 6693 webserver.cc:460] Webserver started at http://127.25.109.2:45997/ using document root <none> and password file <none>
I20260924 16:04:29.410678 6693 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:29.410737 6693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 16:04:29 dist-test-slave-n106 krb5kdc[5469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265869, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for HTTP/127.25.109.20@KRBTEST.COM
I20260924 16:04:29.431674 6693 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:04:29.434605 6709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:29.436223 6693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:04:29.436347 6693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "f179660025c54890ab5560d79c492041"
format_stamp: "Formatted at 2026-09-24 16:04:28 on dist-test-slave-n106"
server_key: "6c8bd8e884d180093796f395f08b23f1"
server_key_iv: "8353b35bb08e1e113ad652383b00bf4a"
server_key_version: "kuduclusterkey@0"
I20260924 16:04:29.437100 6693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:29.466714 6693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:29.470496 6693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:29.470775 6693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:29.471503 6693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:29.472600 6693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:29.472677 6693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:29.472740 6693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:29.472774 6693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:29.484731 6693 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:39517
I20260924 16:04:29.486043 6693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:04:29.492552 6824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:39517 every 8 connection(s)
I20260924 16:04:29.494164 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 6693
I20260924 16:04:29.494656 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.109.20:37853/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:42129
--local_ip_for_outbound_sockets=127.25.109.3
--tserver_master_addrs=127.25.109.62:32817
--webserver_port=36919
--webserver_interface=127.25.109.3
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.25.109.20:37853/kms with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:04:29 dist-test-slave-n106 krb5kdc[5469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265869, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:04:29.504325 6828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:29.488757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:32817 (local address 127.25.109.2:37009)
0924 16:04:29.489319 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:29.489377 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:29.490504 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:29.491180 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:29.491190 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:29.491546 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:29.492244 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:29.492266 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.493424 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.493427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:29.494891 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:29.494903 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.495731 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.495735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:29.495940 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:29.496767 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:29.496788 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:29.498533 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:29.500722 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:29.500728 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:29.500745 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:29.501078 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:29.501631 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:29.501635 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:29.501637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:29.501816 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:29.502311 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:29.502318 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:29.502658 (+ 340us) client_negotiation.cc:769] Sending connection context
0924 16:04:29.502883 (+ 225us) client_negotiation.cc:240] Negotiation successful
0924 16:04:29.503224 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":143,"threads_started":1}
I20260924 16:04:29.505071 6692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:29.488206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:37009 (local address 127.25.109.62:32817)
0924 16:04:29.488343 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:29.488348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:29.489488 (+ 1140us) server_negotiation.cc:407] Connection header received
0924 16:04:29.490771 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:29.490775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:29.490840 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:29.490944 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:29.492459 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.493273 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.495059 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.495416 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.495503 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:29.498735 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:29.498739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:29.498742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:29.498771 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:29.500559 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:29.501407 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:29.501411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:29.501413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:29.501505 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:29.501930 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:29.501934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:29.501936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:29.502156 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:29.502255 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:29.504795 (+ 2540us) server_negotiation.cc:299] Negotiation successful
0924 16:04:29.504920 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 16:04:29.506812 6826 heartbeater.cc:347] Connected to a master server at 127.25.109.62:32817
I20260924 16:04:29.507090 6826 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:29.507669 6826 heartbeater.cc:511] Master 127.25.109.62:32817 requested a full tablet report, sending...
W20260924 16:04:29.509384 6826 heartbeater.cc:644] Failed to heartbeat to 127.25.109.62:32817 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 16:04:29.509512 6826 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:29.509673 6826 heartbeater.cc:511] Master 127.25.109.62:32817 requested a full tablet report, sending...
I20260924 16:04:29.510519 6826 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:29.510706 6826 heartbeater.cc:511] Master 127.25.109.62:32817 requested a full tablet report, sending...
W20260924 16:04:29.511535 6826 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 16:04:29.637862 6829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:29.638095 6829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:29.638135 6829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:29.642530 6829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:29.642602 6829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:29.642715 6829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:04:29.648268 6829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.25.109.20:37853/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:42129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=36919
--tserver_master_addrs=127.25.109.62:32817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:04:29.649279 6829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:29.650324 6829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:29.660874 6836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:29.660934 6838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:29.661134 6835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:29.661298 6829 server_base.cc:1034] running on GCE node
I20260924 16:04:29.661718 6829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:29.662500 6829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:29.665834 6829 hybrid_clock.cc:648] HybridClock initialized: now 1790265869665796 us; error 53 us; skew 500 ppm
Sep 24 16:04:29 dist-test-slave-n106 krb5kdc[5469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265869, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:29.668993 6829 init.cc:377] Logged in from keytab as kudu/127.25.109.3@KRBTEST.COM (short username kudu)
I20260924 16:04:29.670176 6829 webserver.cc:460] Webserver started at http://127.25.109.3:36919/ using document root <none> and password file <none>
I20260924 16:04:29.670943 6829 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:29.671067 6829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 16:04:29 dist-test-slave-n106 krb5kdc[5469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265869, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for HTTP/127.25.109.20@KRBTEST.COM
I20260924 16:04:29.706146 6829 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.000s
I20260924 16:04:29.715068 6846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:29.717053 6829 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.004s sys 0.000s
I20260924 16:04:29.717265 6829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "a1d97c6104bb49248d80d5999a11375d"
format_stamp: "Formatted at 2026-09-24 16:04:28 on dist-test-slave-n106"
server_key: "df32f031628b0c516b4e23e2407cc2d1"
server_key_iv: "9d6ce8fcb4002811eb0390f6a64a12b9"
server_key_version: "kuduclusterkey@0"
I20260924 16:04:29.718132 6829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:29.746785 6829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:29.750415 6829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:29.750638 6829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:29.751386 6829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:29.752478 6829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:29.752596 6829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:29.752724 6829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:29.752774 6829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:29.764717 6829 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:42129
I20260924 16:04:29.764806 6970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:42129 every 8 connection(s)
I20260924 16:04:29.766078 6829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:04:29.774775 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 6829
Sep 24 16:04:29 dist-test-slave-n106 krb5kdc[5469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265869, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:04:29.783756 6692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:29.768101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:33603 (local address 127.25.109.62:32817)
0924 16:04:29.768239 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:29.768243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:29.769400 (+ 1157us) server_negotiation.cc:407] Connection header received
0924 16:04:29.770703 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:29.770707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:29.770765 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:29.770861 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:29.772341 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.773208 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.774935 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.775260 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.775329 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:29.779632 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:29.779636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:29.779639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:29.779664 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:29.781312 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:29.781973 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:29.781977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:29.781979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:29.782075 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:29.782511 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:29.782515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:29.782517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:29.782734 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:29.782842 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:29.783452 (+ 610us) server_negotiation.cc:299] Negotiation successful
0924 16:04:29.783583 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 16:04:29.784730 6973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:29.768675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:32817 (local address 127.25.109.3:33603)
0924 16:04:29.769231 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:29.769289 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:29.770416 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:29.771080 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:29.771092 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:29.771458 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:29.772147 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:29.772168 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.773372 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.773375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:29.774775 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:29.774789 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.775585 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.775591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:29.775804 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:29.777879 (+ 2075us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:29.777900 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:29.779440 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:29.781445 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:29.781450 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:29.781467 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:29.781846 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:29.782195 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:29.782198 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:29.782200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:29.782388 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:29.782859 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:29.782868 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:29.783150 (+ 282us) client_negotiation.cc:769] Sending connection context
0924 16:04:29.783364 (+ 214us) client_negotiation.cc:240] Negotiation successful
0924 16:04:29.783691 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":132,"threads_started":1}
I20260924 16:04:29.786065 6971 heartbeater.cc:347] Connected to a master server at 127.25.109.62:32817
I20260924 16:04:29.786336 6971 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:29.786916 6971 heartbeater.cc:511] Master 127.25.109.62:32817 requested a full tablet report, sending...
I20260924 16:04:29.788507 6484 ts_manager.cc:194] Registered new tserver with Master: a1d97c6104bb49248d80d5999a11375d (127.25.109.3:42129)
I20260924 16:04:29.788968 6975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:29.776497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52224 (local address 127.25.109.62:32817)
0924 16:04:29.776784 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:29.776790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:29.776802 (+ 12us) server_negotiation.cc:407] Connection header received
0924 16:04:29.776832 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:29.776834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:29.776870 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:29.776974 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:29.777753 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.778540 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.780345 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:29.780720 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:29.780847 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:29.782974 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:29.782978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:29.782980 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:29.783004 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:29.785214 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:29.787539 (+ 2325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:29.787543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:29.787545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:29.787647 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:29.788001 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:29.788005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:29.788007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:29.788199 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:29.788387 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:29.788651 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 16:04:29.788778 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":119,"threads_started":1}
I20260924 16:04:29.790139 6484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.3@KRBTEST.COM'} at 127.25.109.3:33603
Sep 24 16:04:30 dist-test-slave-n106 krb5kdc[5469](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790265870, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:30.230350 6688 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:30.230672 6688 heartbeater.cc:511] Master 127.25.109.62:32817 requested a full tablet report, sending...
I20260924 16:04:30.231523 6484 ts_manager.cc:194] Registered new tserver with Master: 5df65559974a4de0b981bc7581fcbe79 (127.25.109.1:37067)
I20260924 16:04:30.232322 6484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.1@KRBTEST.COM'} at 127.25.109.1:45819
Sep 24 16:04:30 dist-test-slave-n106 krb5kdc[5469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265870, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for HTTP/127.25.109.20@KRBTEST.COM
I20260924 16:04:30.511783 6826 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:30.512084 6826 heartbeater.cc:511] Master 127.25.109.62:32817 requested a full tablet report, sending...
I20260924 16:04:30.512863 6484 ts_manager.cc:194] Registered new tserver with Master: f179660025c54890ab5560d79c492041 (127.25.109.2:39517)
I20260924 16:04:30.513612 6484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.2@KRBTEST.COM'} at 127.25.109.2:37009
I20260924 16:04:30.515250 26036 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:04:30.526649 6986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:30.516947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52226 (local address 127.25.109.62:32817)
0924 16:04:30.517211 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:30.517215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:30.517478 (+ 263us) server_negotiation.cc:407] Connection header received
0924 16:04:30.517521 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:30.517524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:30.517582 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:30.517664 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:30.518570 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.519379 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.520602 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.520961 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.521080 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:30.522747 (+ 1667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:30.522751 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:30.522754 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:30.522782 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:30.524755 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:30.525240 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:30.525244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:30.525246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:30.525345 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:30.525635 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:30.525638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:30.525640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:30.525942 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:30.526077 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:30.526317 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 16:04:30.526457 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":92,"threads_started":1}
I20260924 16:04:30.531279 6484 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:04:30.536614 6484 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:04:30.555990 6991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:30.548023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:42129 (local address 127.0.0.1:44954)
0924 16:04:30.548997 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:30.549057 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:30.549181 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:30.550061 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:30.550067 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:30.550089 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:30.550213 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:30.550219 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.552142 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.552145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:30.554202 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:30.554213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.555460 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.555463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:30.555570 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:30.555608 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 16:04:30.555720 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 16:04:30.555830 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"thread_start_us":99,"threads_started":1}
I20260924 16:04:30.557359 6992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:30.548175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44954 (local address 127.25.109.3:42129)
0924 16:04:30.549451 (+ 1276us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:30.549458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:30.549481 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:04:30.549554 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:30.549559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:30.549753 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:30.549889 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:30.550853 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.552008 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.554338 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.555138 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.555218 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:30.555365 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:30.557109 (+ 1744us) server_negotiation.cc:299] Negotiation successful
0924 16:04:30.557217 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1167,"thread_start_us":76,"threads_started":1}
I20260924 16:04:30.562166 6995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:30.549058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:37067 (local address 127.0.0.1:48702)
0924 16:04:30.550386 (+ 1328us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:30.550399 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:30.550479 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:30.552864 (+ 2385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:30.552867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:30.552883 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:30.552969 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:30.552975 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.554310 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.554313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:30.555347 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:30.555359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.561699 (+ 6340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.561703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:30.561805 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:30.561832 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 16:04:30.561942 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 16:04:30.562020 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1272,"thread_start_us":86,"threads_started":1}
I20260924 16:04:30.561755 6896 tablet_service.cc:1467] Processing CreateTablet for tablet 7d5074c876f448b19517fdb88d76a8ff (DEFAULT_TABLE table=test-table [id=ac3ae4e966064bbc9b4a9294263266b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:04:30.563062 6896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d5074c876f448b19517fdb88d76a8ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:30.563735 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:30.549228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48702 (local address 127.25.109.1:37067)
0924 16:04:30.552298 (+ 3070us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:30.552304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:30.552324 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:04:30.552394 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:30.552399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:30.552601 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:30.552735 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:30.553106 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.554181 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.556576 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.561359 (+ 4783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.561462 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:30.561607 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:30.563480 (+ 1873us) server_negotiation.cc:299] Negotiation successful
0924 16:04:30.563594 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2970,"thread_start_us":54,"threads_started":1}
I20260924 16:04:30.565867 6993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:30.549666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:39517 (local address 127.0.0.1:47574)
0924 16:04:30.555564 (+ 5898us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:30.555577 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:30.555659 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:30.556330 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:30.556333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:30.556350 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:30.556446 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:30.556452 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.560341 (+ 3889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.560345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:30.564104 (+ 3759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:30.564116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.565477 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.565480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:30.565571 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:30.565596 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 16:04:30.565658 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 16:04:30.565725 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5842,"thread_start_us":74,"threads_started":1}
I20260924 16:04:30.566632 6996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:30.550014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47574 (local address 127.25.109.2:39517)
0924 16:04:30.550691 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:30.550697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:30.555751 (+ 5054us) server_negotiation.cc:407] Connection header received
0924 16:04:30.555846 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:30.555851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:30.556027 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:30.556174 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:30.559101 (+ 2927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.560192 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.564255 (+ 4063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.565168 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.565277 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:30.565413 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:30.566387 (+ 974us) server_negotiation.cc:299] Negotiation successful
0924 16:04:30.566493 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":94,"threads_started":1}
I20260924 16:04:30.568982 6605 tablet_service.cc:1467] Processing CreateTablet for tablet 7d5074c876f448b19517fdb88d76a8ff (DEFAULT_TABLE table=test-table [id=ac3ae4e966064bbc9b4a9294263266b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:04:30.569459 6741 tablet_service.cc:1467] Processing CreateTablet for tablet 7d5074c876f448b19517fdb88d76a8ff (DEFAULT_TABLE table=test-table [id=ac3ae4e966064bbc9b4a9294263266b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:04:30.570082 6605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d5074c876f448b19517fdb88d76a8ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:30.570475 6741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d5074c876f448b19517fdb88d76a8ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:30.580837 6999 tablet_bootstrap.cc:492] T 7d5074c876f448b19517fdb88d76a8ff P f179660025c54890ab5560d79c492041: Bootstrap starting.
I20260924 16:04:30.581136 6997 tablet_bootstrap.cc:492] T 7d5074c876f448b19517fdb88d76a8ff P a1d97c6104bb49248d80d5999a11375d: Bootstrap starting.
I20260924 16:04:30.583359 6997 tablet_bootstrap.cc:654] T 7d5074c876f448b19517fdb88d76a8ff P a1d97c6104bb49248d80d5999a11375d: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:30.583565 6999 tablet_bootstrap.cc:654] T 7d5074c876f448b19517fdb88d76a8ff P f179660025c54890ab5560d79c492041: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:30.584300 6997 log.cc:824] T 7d5074c876f448b19517fdb88d76a8ff P a1d97c6104bb49248d80d5999a11375d: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:30.584476 6999 log.cc:824] T 7d5074c876f448b19517fdb88d76a8ff P f179660025c54890ab5560d79c492041: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:30.585057 6998 tablet_bootstrap.cc:492] T 7d5074c876f448b19517fdb88d76a8ff P 5df65559974a4de0b981bc7581fcbe79: Bootstrap starting.
I20260924 16:04:30.586388 6997 tablet_bootstrap.cc:492] T 7d5074c876f448b19517fdb88d76a8ff P a1d97c6104bb49248d80d5999a11375d: No bootstrap required, opened a new log
I20260924 16:04:30.586565 6997 ts_tablet_manager.cc:1397] T 7d5074c876f448b19517fdb88d76a8ff P a1d97c6104bb49248d80d5999a11375d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 16:04:30.591051 6998 tablet_bootstrap.cc:654] T 7d5074c876f448b19517fdb88d76a8ff P 5df65559974a4de0b981bc7581fcbe79: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:30.590961 6997 raft_consensus.cc:348] T 7d5074c876f448b19517fdb88d76a8ff P a1d97c6104bb49248d80d5999a11375d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1d97c6104bb49248d80d5999a11375d" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42129 } } peers { permanent_uuid: "5df65559974a4de0b981bc7581fcbe79" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37067 } } peers { permanent_uuid: "f179660025c54890ab5560d79c492041" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 39517 } }
I20260924 16:04:30.591224 6997 raft_consensus.cc:374] T 7d5074c876f448b19517fdb88d76a8ff P a1d97c6104bb49248d80d5999a11375d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:30.591274 6997 raft_consensus.cc:729] T 7d5074c876f448b19517fdb88d76a8ff P a1d97c6104bb49248d80d5999a11375d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1d97c6104bb49248d80d5999a11375d, State: Initialized, Role: FOLLOWER
I20260924 16:04:30.591894 6997 consensus_queue.cc:260] T 7d5074c876f448b19517fdb88d76a8ff P a1d97c6104bb49248d80d5999a11375d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1d97c6104bb49248d80d5999a11375d" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42129 } } peers { permanent_uuid: "5df65559974a4de0b981bc7581fcbe79" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37067 } } peers { permanent_uuid: "f179660025c54890ab5560d79c492041" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 39517 } }
I20260924 16:04:30.591944 6998 log.cc:824] T 7d5074c876f448b19517fdb88d76a8ff P 5df65559974a4de0b981bc7581fcbe79: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:30.592994 6997 ts_tablet_manager.cc:1428] T 7d5074c876f448b19517fdb88d76a8ff P a1d97c6104bb49248d80d5999a11375d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 16:04:30.593393 6971 heartbeater.cc:503] Master 127.25.109.62:32817 was elected leader, sending a full tablet report...
I20260924 16:04:30.594980 6998 tablet_bootstrap.cc:492] T 7d5074c876f448b19517fdb88d76a8ff P 5df65559974a4de0b981bc7581fcbe79: No bootstrap required, opened a new log
I20260924 16:04:30.595171 6998 ts_tablet_manager.cc:1397] T 7d5074c876f448b19517fdb88d76a8ff P 5df65559974a4de0b981bc7581fcbe79: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260924 16:04:30.599385 6998 raft_consensus.cc:348] T 7d5074c876f448b19517fdb88d76a8ff P 5df65559974a4de0b981bc7581fcbe79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1d97c6104bb49248d80d5999a11375d" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42129 } } peers { permanent_uuid: "5df65559974a4de0b981bc7581fcbe79" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37067 } } peers { permanent_uuid: "f179660025c54890ab5560d79c492041" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 39517 } }
I20260924 16:04:30.599629 6999 tablet_bootstrap.cc:492] T 7d5074c876f448b19517fdb88d76a8ff P f179660025c54890ab5560d79c492041: No bootstrap required, opened a new log
I20260924 16:04:30.599718 6998 raft_consensus.cc:374] T 7d5074c876f448b19517fdb88d76a8ff P 5df65559974a4de0b981bc7581fcbe79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:30.599807 6998 raft_consensus.cc:729] T 7d5074c876f448b19517fdb88d76a8ff P 5df65559974a4de0b981bc7581fcbe79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5df65559974a4de0b981bc7581fcbe79, State: Initialized, Role: FOLLOWER
I20260924 16:04:30.599903 6999 ts_tablet_manager.cc:1397] T 7d5074c876f448b19517fdb88d76a8ff P f179660025c54890ab5560d79c492041: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20260924 16:04:30.600317 6998 consensus_queue.cc:260] T 7d5074c876f448b19517fdb88d76a8ff P 5df65559974a4de0b981bc7581fcbe79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1d97c6104bb49248d80d5999a11375d" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42129 } } peers { permanent_uuid: "5df65559974a4de0b981bc7581fcbe79" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37067 } } peers { permanent_uuid: "f179660025c54890ab5560d79c492041" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 39517 } }
I20260924 16:04:30.601482 6998 ts_tablet_manager.cc:1428] T 7d5074c876f448b19517fdb88d76a8ff P 5df65559974a4de0b981bc7581fcbe79: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260924 16:04:30.601889 6688 heartbeater.cc:503] Master 127.25.109.62:32817 was elected leader, sending a full tablet report...
I20260924 16:04:30.605012 6999 raft_consensus.cc:348] T 7d5074c876f448b19517fdb88d76a8ff P f179660025c54890ab5560d79c492041 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1d97c6104bb49248d80d5999a11375d" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42129 } } peers { permanent_uuid: "5df65559974a4de0b981bc7581fcbe79" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37067 } } peers { permanent_uuid: "f179660025c54890ab5560d79c492041" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 39517 } }
I20260924 16:04:30.605388 6999 raft_consensus.cc:374] T 7d5074c876f448b19517fdb88d76a8ff P f179660025c54890ab5560d79c492041 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:30.605482 6999 raft_consensus.cc:729] T 7d5074c876f448b19517fdb88d76a8ff P f179660025c54890ab5560d79c492041 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f179660025c54890ab5560d79c492041, State: Initialized, Role: FOLLOWER
I20260924 16:04:30.605963 6999 consensus_queue.cc:260] T 7d5074c876f448b19517fdb88d76a8ff P f179660025c54890ab5560d79c492041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1d97c6104bb49248d80d5999a11375d" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42129 } } peers { permanent_uuid: "5df65559974a4de0b981bc7581fcbe79" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37067 } } peers { permanent_uuid: "f179660025c54890ab5560d79c492041" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 39517 } }
I20260924 16:04:30.606680 6826 heartbeater.cc:503] Master 127.25.109.62:32817 was elected leader, sending a full tablet report...
I20260924 16:04:30.607053 6999 ts_tablet_manager.cc:1428] T 7d5074c876f448b19517fdb88d76a8ff P f179660025c54890ab5560d79c492041: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260924 16:04:30.697759 6689 tablet.cc:2369] T 7d5074c876f448b19517fdb88d76a8ff P 5df65559974a4de0b981bc7581fcbe79: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 16:04:30 dist-test-slave-n106 krb5kdc[5469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265870, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for HTTP/127.25.109.20@KRBTEST.COM
W20260924 16:04:30.739051 6827 tablet.cc:2369] T 7d5074c876f448b19517fdb88d76a8ff P f179660025c54890ab5560d79c492041: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:04:30.767712 6972 tablet.cc:2369] T 7d5074c876f448b19517fdb88d76a8ff P a1d97c6104bb49248d80d5999a11375d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:04:30.789672 7004 raft_consensus.cc:482] T 7d5074c876f448b19517fdb88d76a8ff P 5df65559974a4de0b981bc7581fcbe79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:04:30.789913 7004 raft_consensus.cc:504] T 7d5074c876f448b19517fdb88d76a8ff P 5df65559974a4de0b981bc7581fcbe79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1d97c6104bb49248d80d5999a11375d" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42129 } } peers { permanent_uuid: "5df65559974a4de0b981bc7581fcbe79" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37067 } } peers { permanent_uuid: "f179660025c54890ab5560d79c492041" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 39517 } }
I20260924 16:04:30.791291 7004 leader_election.cc:288] T 7d5074c876f448b19517fdb88d76a8ff P 5df65559974a4de0b981bc7581fcbe79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1d97c6104bb49248d80d5999a11375d (127.25.109.3:42129), f179660025c54890ab5560d79c492041 (127.25.109.2:39517)
I20260924 16:04:30.796615 6992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:30.791958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:38881 (local address 127.25.109.3:42129)
0924 16:04:30.792092 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:30.792097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:30.792418 (+ 321us) server_negotiation.cc:407] Connection header received
0924 16:04:30.792637 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:30.792641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:30.792686 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:30.792729 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:30.793116 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.793858 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.795381 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.795919 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.796009 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:30.796062 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:30.796422 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 16:04:30.796496 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 16:04:30.797209 7009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:30.791524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:42129 (local address 127.25.109.1:38881)
0924 16:04:30.792339 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:30.792352 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:30.792517 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:30.792858 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:30.792861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:30.792876 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:30.793007 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:30.793012 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.793999 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.794002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:30.795242 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:30.795253 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.796170 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.796174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:30.796270 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:30.796293 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 16:04:30.796983 (+ 690us) client_negotiation.cc:240] Negotiation successful
0924 16:04:30.797061 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"thread_start_us":127,"threads_started":1}
I20260924 16:04:30.798086 6899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d5074c876f448b19517fdb88d76a8ff" candidate_uuid: "5df65559974a4de0b981bc7581fcbe79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1d97c6104bb49248d80d5999a11375d" is_pre_election: true
I20260924 16:04:30.798436 6899 raft_consensus.cc:2454] T 7d5074c876f448b19517fdb88d76a8ff P a1d97c6104bb49248d80d5999a11375d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5df65559974a4de0b981bc7581fcbe79 in term 0.
I20260924 16:04:30.799072 6572 leader_election.cc:302] T 7d5074c876f448b19517fdb88d76a8ff P 5df65559974a4de0b981bc7581fcbe79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5df65559974a4de0b981bc7581fcbe79, a1d97c6104bb49248d80d5999a11375d; no voters:
I20260924 16:04:30.799377 7004 raft_consensus.cc:2781] T 7d5074c876f448b19517fdb88d76a8ff P 5df65559974a4de0b981bc7581fcbe79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:04:30.799471 7004 raft_consensus.cc:482] T 7d5074c876f448b19517fdb88d76a8ff P 5df65559974a4de0b981bc7581fcbe79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:04:30.799511 7004 raft_consensus.cc:3037] T 7d5074c876f448b19517fdb88d76a8ff P 5df65559974a4de0b981bc7581fcbe79 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:30.800880 7004 raft_consensus.cc:504] T 7d5074c876f448b19517fdb88d76a8ff P 5df65559974a4de0b981bc7581fcbe79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1d97c6104bb49248d80d5999a11375d" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42129 } } peers { permanent_uuid: "5df65559974a4de0b981bc7581fcbe79" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37067 } } peers { permanent_uuid: "f179660025c54890ab5560d79c492041" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 39517 } }
I20260924 16:04:30.801237 7004 leader_election.cc:288] T 7d5074c876f448b19517fdb88d76a8ff P 5df65559974a4de0b981bc7581fcbe79 [CANDIDATE]: Term 1 election: Requested vote from peers a1d97c6104bb49248d80d5999a11375d (127.25.109.3:42129), f179660025c54890ab5560d79c492041 (127.25.109.2:39517)
I20260924 16:04:30.801780 6899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d5074c876f448b19517fdb88d76a8ff" candidate_uuid: "5df65559974a4de0b981bc7581fcbe79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1d97c6104bb49248d80d5999a11375d"
I20260924 16:04:30.801908 6899 raft_consensus.cc:3037] T 7d5074c876f448b19517fdb88d76a8ff P a1d97c6104bb49248d80d5999a11375d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:30.802033 6996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:30.792116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:51817 (local address 127.25.109.2:39517)
0924 16:04:30.792288 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:30.792293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:30.797732 (+ 5439us) server_negotiation.cc:407] Connection header received
0924 16:04:30.797912 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:30.797916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:30.797964 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:30.798032 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:30.798376 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.799078 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.800519 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.801163 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.801254 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:30.801322 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:30.801835 (+ 513us) server_negotiation.cc:299] Negotiation successful
0924 16:04:30.801915 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 16:04:30.802490 7008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:30.791630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:39517 (local address 127.25.109.1:51817)
0924 16:04:30.797634 (+ 6004us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:30.797649 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:30.797807 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:30.798140 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:30.798143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:30.798159 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:30.798259 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:30.798264 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.799228 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.799231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:30.800355 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:30.800366 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.801528 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.801532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:30.801675 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:30.801699 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 16:04:30.802291 (+ 592us) client_negotiation.cc:240] Negotiation successful
0924 16:04:30.802361 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5937,"thread_start_us":101,"threads_started":1}
I20260924 16:04:30.803179 6899 raft_consensus.cc:2454] T 7d5074c876f448b19517fdb88d76a8ff P a1d97c6104bb49248d80d5999a11375d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5df65559974a4de0b981bc7581fcbe79 in term 1.
I20260924 16:04:30.803149 6760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d5074c876f448b19517fdb88d76a8ff" candidate_uuid: "5df65559974a4de0b981bc7581fcbe79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f179660025c54890ab5560d79c492041" is_pre_election: true
I20260924 16:04:30.803196 6778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d5074c876f448b19517fdb88d76a8ff" candidate_uuid: "5df65559974a4de0b981bc7581fcbe79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f179660025c54890ab5560d79c492041"
I20260924 16:04:30.803372 6778 raft_consensus.cc:3037] T 7d5074c876f448b19517fdb88d76a8ff P f179660025c54890ab5560d79c492041 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:30.803539 6572 leader_election.cc:302] T 7d5074c876f448b19517fdb88d76a8ff P 5df65559974a4de0b981bc7581fcbe79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5df65559974a4de0b981bc7581fcbe79, a1d97c6104bb49248d80d5999a11375d; no voters:
I20260924 16:04:30.803701 7004 raft_consensus.cc:2781] T 7d5074c876f448b19517fdb88d76a8ff P 5df65559974a4de0b981bc7581fcbe79 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:30.803921 7004 raft_consensus.cc:686] T 7d5074c876f448b19517fdb88d76a8ff P 5df65559974a4de0b981bc7581fcbe79 [term 1 LEADER]: Becoming Leader. State: Replica: 5df65559974a4de0b981bc7581fcbe79, State: Running, Role: LEADER
I20260924 16:04:30.804275 7004 consensus_queue.cc:237] T 7d5074c876f448b19517fdb88d76a8ff P 5df65559974a4de0b981bc7581fcbe79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1d97c6104bb49248d80d5999a11375d" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42129 } } peers { permanent_uuid: "5df65559974a4de0b981bc7581fcbe79" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37067 } } peers { permanent_uuid: "f179660025c54890ab5560d79c492041" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 39517 } }
I20260924 16:04:30.804768 6778 raft_consensus.cc:2454] T 7d5074c876f448b19517fdb88d76a8ff P f179660025c54890ab5560d79c492041 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5df65559974a4de0b981bc7581fcbe79 in term 1.
I20260924 16:04:30.807483 6484 catalog_manager.cc:5495] T 7d5074c876f448b19517fdb88d76a8ff P 5df65559974a4de0b981bc7581fcbe79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5df65559974a4de0b981bc7581fcbe79 (127.25.109.1). New cstate: current_term: 1 leader_uuid: "5df65559974a4de0b981bc7581fcbe79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1d97c6104bb49248d80d5999a11375d" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5df65559974a4de0b981bc7581fcbe79" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 37067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f179660025c54890ab5560d79c492041" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 39517 } health_report { overall_health: UNKNOWN } } }
I20260924 16:04:30.853338 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:30.848061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48708 (local address 127.25.109.1:37067)
0924 16:04:30.848187 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:30.848192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:30.848206 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:04:30.848299 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:30.848302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:30.848378 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:30.848517 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:04:30.848953 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.849738 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.851045 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:30.851375 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:30.851477 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:30.851900 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:04:30.852529 (+ 629us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:04:30.852653 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:30.853102 (+ 449us) server_negotiation.cc:299] Negotiation successful
0924 16:04:30.853166 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 16:04:30.862187 6778 raft_consensus.cc:1261] T 7d5074c876f448b19517fdb88d76a8ff P f179660025c54890ab5560d79c492041 [term 1 FOLLOWER]: Refusing update from remote peer 5df65559974a4de0b981bc7581fcbe79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:04:30.863130 7010 consensus_queue.cc:1035] T 7d5074c876f448b19517fdb88d76a8ff P 5df65559974a4de0b981bc7581fcbe79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f179660025c54890ab5560d79c492041" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 39517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:04:30.864441 6899 raft_consensus.cc:1261] T 7d5074c876f448b19517fdb88d76a8ff P a1d97c6104bb49248d80d5999a11375d [term 1 FOLLOWER]: Refusing update from remote peer 5df65559974a4de0b981bc7581fcbe79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:04:30.865348 7010 consensus_queue.cc:1035] T 7d5074c876f448b19517fdb88d76a8ff P 5df65559974a4de0b981bc7581fcbe79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1d97c6104bb49248d80d5999a11375d" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 42129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:04:30.877333 7016 mvcc.cc:205] Tried to move back new op lower bound from 7332929007040380928 to 7332929006817550336. Current Snapshot: MvccSnapshot[applied={T|T < 7332929007040380928}]
I20260924 16:04:30.877579 7013 mvcc.cc:205] Tried to move back new op lower bound from 7332929007040380928 to 7332929006817550336. Current Snapshot: MvccSnapshot[applied={T|T < 7332929007040380928}]
I20260924 16:04:30.880524 7014 mvcc.cc:205] Tried to move back new op lower bound from 7332929007040380928 to 7332929006817550336. Current Snapshot: MvccSnapshot[applied={T|T < 7332929007040380928}]
I20260924 16:04:30.889571 6484 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:04:30.889748 6484 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:04:30.892335 6484 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 931a69552f774dd18e9a7ac45276270d: Sending DeleteTablet for 3 replicas of tablet 7d5074c876f448b19517fdb88d76a8ff
I20260924 16:04:30.893580 6896 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d5074c876f448b19517fdb88d76a8ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:30 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:44954
I20260924 16:04:30.894104 6741 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d5074c876f448b19517fdb88d76a8ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:30 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:47574
I20260924 16:04:30.894860 6605 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d5074c876f448b19517fdb88d76a8ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:30 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:48702
I20260924 16:04:30.896147 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 6524
I20260924 16:04:30.901458 7023 tablet_replica.cc:330] T 7d5074c876f448b19517fdb88d76a8ff P f179660025c54890ab5560d79c492041: stopping tablet replica
I20260924 16:04:30.901844 7023 raft_consensus.cc:2229] T 7d5074c876f448b19517fdb88d76a8ff P f179660025c54890ab5560d79c492041 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:04:30.902298 7023 raft_consensus.cc:2258] T 7d5074c876f448b19517fdb88d76a8ff P f179660025c54890ab5560d79c492041 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:04:30.902714 7022 tablet_replica.cc:330] T 7d5074c876f448b19517fdb88d76a8ff P a1d97c6104bb49248d80d5999a11375d: stopping tablet replica
I20260924 16:04:30.903008 7022 raft_consensus.cc:2229] T 7d5074c876f448b19517fdb88d76a8ff P a1d97c6104bb49248d80d5999a11375d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:04:30.903276 7022 raft_consensus.cc:2258] T 7d5074c876f448b19517fdb88d76a8ff P a1d97c6104bb49248d80d5999a11375d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:04:30.909449 7023 ts_tablet_manager.cc:1905] T 7d5074c876f448b19517fdb88d76a8ff P f179660025c54890ab5560d79c492041: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:04:30.911267 7022 ts_tablet_manager.cc:1905] T 7d5074c876f448b19517fdb88d76a8ff P a1d97c6104bb49248d80d5999a11375d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:04:30.914016 7022 ts_tablet_manager.cc:1918] T 7d5074c876f448b19517fdb88d76a8ff P a1d97c6104bb49248d80d5999a11375d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:04:30.914180 7022 log.cc:1196] T 7d5074c876f448b19517fdb88d76a8ff P a1d97c6104bb49248d80d5999a11375d: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-2/wal/wals/7d5074c876f448b19517fdb88d76a8ff
I20260924 16:04:30.914572 7022 ts_tablet_manager.cc:1939] T 7d5074c876f448b19517fdb88d76a8ff P a1d97c6104bb49248d80d5999a11375d: Deleting consensus metadata
I20260924 16:04:30.915578 7023 ts_tablet_manager.cc:1918] T 7d5074c876f448b19517fdb88d76a8ff P f179660025c54890ab5560d79c492041: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:04:30.915747 7023 log.cc:1196] T 7d5074c876f448b19517fdb88d76a8ff P f179660025c54890ab5560d79c492041: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265708917108-26036-0/minicluster-data/ts-1/wal/wals/7d5074c876f448b19517fdb88d76a8ff
I20260924 16:04:30.915951 6468 catalog_manager.cc:4841] TS a1d97c6104bb49248d80d5999a11375d (127.25.109.3:42129): tablet 7d5074c876f448b19517fdb88d76a8ff (table test-table [id=ac3ae4e966064bbc9b4a9294263266b4]) successfully deleted
I20260924 16:04:30.916173 7023 ts_tablet_manager.cc:1939] T 7d5074c876f448b19517fdb88d76a8ff P f179660025c54890ab5560d79c492041: Deleting consensus metadata
W20260924 16:04:30.916491 6470 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.109.1:37067 (error 108)
I20260924 16:04:30.917450 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 6693
I20260924 16:04:30.917449 6470 catalog_manager.cc:4841] TS f179660025c54890ab5560d79c492041 (127.25.109.2:39517): tablet 7d5074c876f448b19517fdb88d76a8ff (table test-table [id=ac3ae4e966064bbc9b4a9294263266b4]) successfully deleted
I20260924 16:04:30.925495 6993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:30.925133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:37067 (local address 127.0.0.1:48714)
0924 16:04:30.925331 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:30.925409 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.1:37067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260924 16:04:30.925806 6470 catalog_manager.cc:4584] TS 5df65559974a4de0b981bc7581fcbe79 (127.25.109.1:37067): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7d5074c876f448b19517fdb88d76a8ff: Network error: Client connection negotiation failed: client connection to 127.25.109.1:37067: connect: Connection refused (error 111)
I20260924 16:04:30.927739 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 6829
I20260924 16:04:30.937100 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 6451
I20260924 16:04:30.944684 26036 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 16:04:31.290673 26036 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 16:04:31.290786 26036 mini_ranger.cc:66] Stopping Ranger...
I20260924 16:04:31.635780 26036 mini_ranger.cc:68] Stopped Ranger
2026-09-24 16:04:31.636 UTC [5500] LOG: received smart shutdown request
2026-09-24 16:04:31.640 UTC [5500] LOG: background worker "logical replication launcher" (PID 5510) exited with exit code 1
2026-09-24 16:04:31.640 UTC [5504] LOG: shutting down
2026-09-24 16:04:31.804 UTC [5500] LOG: database system is shut down
2026-09-24T16:04:31Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48214 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:04:31 dist-test-slave-n106 krb5kdc[7034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:04:31 dist-test-slave-n106 krb5kdc[7034](info): set up 2 sockets
Sep 24 16:04:31 dist-test-slave-n106 krb5kdc[7034](info): commencing operation
krb5kdc: starting...
W20260924 16:04:32.965332 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:04:33 dist-test-slave-n106 krb5kdc[7034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:04:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:04:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
I20260924 16:04:33.085484 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:41325
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:40627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:41325,127.25.109.61:40505,127.25.109.60:38417
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:33.196595 7047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:33.196928 7047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:33.197007 7047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:33.201150 7047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:04:33.201236 7047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:33.201273 7047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:33.201331 7047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:04:33.201372 7047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:04:33.206750 7047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:40627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:41325,127.25.109.61:40505,127.25.109.60:38417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:41325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:04:33.207919 7047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:33.209100 7047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:33.217866 7052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:33.217864 7055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:33.217963 7047 server_base.cc:1034] running on GCE node
W20260924 16:04:33.217860 7053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:33.218672 7047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:33.219790 7047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:33.220948 7047 hybrid_clock.cc:648] HybridClock initialized: now 1790265873220917 us; error 43 us; skew 500 ppm
Sep 24 16:04:33 dist-test-slave-n106 krb5kdc[7034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265873, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:33.223587 7047 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:04:33.224599 7047 webserver.cc:460] Webserver started at http://127.25.109.62:34451/ using document root <none> and password file <none>
I20260924 16:04:33.225198 7047 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:33.225296 7047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:33.225546 7047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:33.227658 7047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "e70f5176f8a64d50a5534b205a949fd9"
format_stamp: "Formatted at 2026-09-24 16:04:33 on dist-test-slave-n106"
server_key: "7fa03128fa2c3a3e35a729c616809a18"
server_key_iv: "e6138a1b565492adbcd51da99e67cb70"
server_key_version: "encryptionkey@0"
I20260924 16:04:33.228248 7047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "e70f5176f8a64d50a5534b205a949fd9"
format_stamp: "Formatted at 2026-09-24 16:04:33 on dist-test-slave-n106"
server_key: "7fa03128fa2c3a3e35a729c616809a18"
server_key_iv: "e6138a1b565492adbcd51da99e67cb70"
server_key_version: "encryptionkey@0"
I20260924 16:04:33.232084 7047 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:04:33.234586 7062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:33.235787 7047 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:04:33.235934 7047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "e70f5176f8a64d50a5534b205a949fd9"
format_stamp: "Formatted at 2026-09-24 16:04:33 on dist-test-slave-n106"
server_key: "7fa03128fa2c3a3e35a729c616809a18"
server_key_iv: "e6138a1b565492adbcd51da99e67cb70"
server_key_version: "encryptionkey@0"
I20260924 16:04:33.236059 7047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:33.261016 7047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:33.264248 7047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:33.264519 7047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:33.271834 7047 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:41325
I20260924 16:04:33.271914 7116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:41325 every 8 connection(s)
I20260924 16:04:33.273011 7047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:04:33.275842 7117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:33.279673 7117 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41325 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:04:33.282070 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 7047
I20260924 16:04:33.282208 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:04:33.282539 26036 external_mini_cluster.cc:1428] Setting key 558a1b02d00610141f8d03ec3caab032
Sep 24 16:04:33 dist-test-slave-n106 krb5kdc[7034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265873, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.109.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.61@KRBTEST.COM" created.
I20260924 16:04:33.296386 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.282242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:41325 (local address 127.0.0.1:37106)
0924 16:04:33.282865 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:33.282911 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:33.283893 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:33.284818 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:33.284827 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:33.285262 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:33.285634 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:33.285660 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.286921 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.286927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:33.288025 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:33.288036 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.289043 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.289046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:33.289130 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:33.289817 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:33.289834 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:33.291228 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:33.293216 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:33.293225 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:33.293235 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:33.293510 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:33.293877 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:33.293880 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:33.293882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:33.294021 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:33.294755 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:33.294765 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:33.294949 (+ 184us) client_negotiation.cc:769] Sending connection context
0924 16:04:33.295170 (+ 221us) client_negotiation.cc:240] Negotiation successful
0924 16:04:33.295574 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":117,"thread_start_us":160,"threads_started":1}
I20260924 16:04:33.296392 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.283277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37106 (local address 127.25.109.62:41325)
0924 16:04:33.283638 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:33.283649 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:33.283700 (+ 51us) server_negotiation.cc:407] Connection header received
0924 16:04:33.284256 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:33.284263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:33.284567 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:33.284786 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:33.285868 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.286774 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.288153 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.288874 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.289092 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:33.291399 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:33.291404 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:33.291415 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:33.291435 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:33.293024 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:33.293620 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:33.293626 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:33.293628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:33.293736 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:33.294131 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:33.294137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:33.294142 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:33.294587 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:33.294697 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:33.295233 (+ 536us) server_negotiation.cc:299] Negotiation successful
0924 16:04:33.295457 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":255,"thread_start_us":87,"threads_started":1}
I20260924 16:04:33.298974 7117 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.61" port: 40505 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:04:33.299748 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.299501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:40505 (local address 127.0.0.1:52528)
0924 16:04:33.299622 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:33.299692 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.61:40505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 16:04:33.299937 7063 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.61:40505: connect: Connection refused (error 111)
I20260924 16:04:33.300751 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.300551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:40505 (local address 127.0.0.1:52530)
0924 16:04:33.300653 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:33.300703 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.61:40505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 16:04:33.300884 7117 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.109.61:40505: Network error: Client connection negotiation failed: client connection to 127.25.109.61:40505: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.61.keytab.
Entry for principal kudu/127.25.109.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.61.keytab.
I20260924 16:04:33.311403 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.61:40505
--webserver_interface=127.25.109.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.61.keytab
--principal=kudu/127.25.109.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:40627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:41325,127.25.109.61:40505,127.25.109.60:38417
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:04:33.350126 7117 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.109.61" port: 40505 } attempt: 1
I20260924 16:04:33.351019 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.350713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:40505 (local address 127.0.0.1:52542)
0924 16:04:33.350868 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:33.350942 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.61:40505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 16:04:33.351728 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.351514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:40505 (local address 127.0.0.1:52554)
0924 16:04:33.351623 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:33.351675 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.61:40505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 16:04:33.351858 7117 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.109.61:40505: Network error: Client connection negotiation failed: client connection to 127.25.109.61:40505: connect: Connection refused (error 111)
I20260924 16:04:33.420006 7117 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.109.61" port: 40505 } attempt: 2
I20260924 16:04:33.420857 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.420566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:40505 (local address 127.0.0.1:52564)
0924 16:04:33.420712 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:33.420782 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.61:40505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 16:04:33.421542 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.421339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:40505 (local address 127.0.0.1:52566)
0924 16:04:33.421446 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:33.421495 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.61:40505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 16:04:33.421700 7117 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.109.61:40505: Network error: Client connection negotiation failed: client connection to 127.25.109.61:40505: connect: Connection refused (error 111)
W20260924 16:04:33.424544 7124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:33.424811 7124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:33.424911 7124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:33.428887 7124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:04:33.428959 7124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:33.429006 7124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:33.429049 7124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:04:33.429082 7124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:04:33.434440 7124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:40627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:41325,127.25.109.61:40505,127.25.109.60:38417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.61.keytab
--principal=kudu/127.25.109.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.61:40505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.109.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:04:33.435524 7124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:33.436641 7124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:33.444952 7132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:33.445029 7130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:33.445036 7129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:33.445257 7124 server_base.cc:1034] running on GCE node
I20260924 16:04:33.445942 7124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:33.447098 7124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:33.448288 7124 hybrid_clock.cc:648] HybridClock initialized: now 1790265873448271 us; error 29 us; skew 500 ppm
Sep 24 16:04:33 dist-test-slave-n106 krb5kdc[7034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265873, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:33.451040 7124 init.cc:377] Logged in from keytab as kudu/127.25.109.61@KRBTEST.COM (short username kudu)
I20260924 16:04:33.451908 7124 webserver.cc:460] Webserver started at http://127.25.109.61:44711/ using document root <none> and password file <none>
I20260924 16:04:33.452540 7124 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:33.452633 7124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:33.452858 7124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:33.454945 7124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-1/data/instance:
uuid: "31bd609145e04f2eb4d057535e074336"
format_stamp: "Formatted at 2026-09-24 16:04:33 on dist-test-slave-n106"
server_key: "8106cdcb12283fee07fc84a37db6d9e2"
server_key_iv: "f4137de55cdf32c8e25c55500d7e388e"
server_key_version: "encryptionkey@0"
I20260924 16:04:33.455544 7124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-1/wal/instance:
uuid: "31bd609145e04f2eb4d057535e074336"
format_stamp: "Formatted at 2026-09-24 16:04:33 on dist-test-slave-n106"
server_key: "8106cdcb12283fee07fc84a37db6d9e2"
server_key_iv: "f4137de55cdf32c8e25c55500d7e388e"
server_key_version: "encryptionkey@0"
I20260924 16:04:33.459348 7124 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 16:04:33.461933 7139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:33.463260 7124 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:04:33.463423 7124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-1/wal
uuid: "31bd609145e04f2eb4d057535e074336"
format_stamp: "Formatted at 2026-09-24 16:04:33 on dist-test-slave-n106"
server_key: "8106cdcb12283fee07fc84a37db6d9e2"
server_key_iv: "f4137de55cdf32c8e25c55500d7e388e"
server_key_version: "encryptionkey@0"
I20260924 16:04:33.463546 7124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:33.482247 7124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:33.485476 7124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:33.485692 7124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:33.492856 7124 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.61:40505
I20260924 16:04:33.492880 7193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.61:40505 every 8 connection(s)
I20260924 16:04:33.493844 7124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-1/data/info.pb
I20260924 16:04:33.496696 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:33.497812 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 7124
I20260924 16:04:33.497911 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-1/wal/instance
I20260924 16:04:33.498239 26036 external_mini_cluster.cc:1428] Setting key ab2ce7e1380215c42dd6ae89579cf3c8
I20260924 16:04:33.501245 7194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41325 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.25.109.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.60@KRBTEST.COM" created.
Sep 24 16:04:33 dist-test-slave-n106 krb5kdc[7034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265873, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.61@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:04:33.512851 7117 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.109.61" port: 40505 } attempt: 3
I20260924 16:04:33.515285 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.503263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37108 (local address 127.25.109.62:41325)
0924 16:04:33.503403 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:33.503415 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:33.504548 (+ 1133us) server_negotiation.cc:407] Connection header received
0924 16:04:33.505467 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:33.505470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:33.505568 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:33.505662 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:33.506853 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.507381 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.508545 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.508795 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.508912 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:33.511460 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:33.511464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:33.511466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:33.511512 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:33.512998 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:33.513557 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:33.513564 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:33.513568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:33.513661 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:33.514097 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:33.514103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:33.514108 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:33.514372 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:33.514492 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:33.514998 (+ 506us) server_negotiation.cc:299] Negotiation successful
0924 16:04:33.515118 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 16:04:33.516244 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.503829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:41325 (local address 127.0.0.1:37108)
0924 16:04:33.504349 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:33.504391 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:33.505257 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:33.505833 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:33.505840 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:33.506178 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:33.506706 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:33.506718 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.507528 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.507530 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:33.508372 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:33.508381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.508954 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.508959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:33.509137 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:33.509756 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:33.509773 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:33.511272 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:33.513107 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:33.513114 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:33.513130 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:33.513447 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:33.513776 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:33.513780 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:33.513782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:33.513980 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:33.514484 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:33.514491 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:33.514749 (+ 258us) client_negotiation.cc:769] Sending connection context
0924 16:04:33.514943 (+ 194us) client_negotiation.cc:240] Negotiation successful
0924 16:04:33.515269 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":153,"threads_started":1}
I20260924 16:04:33.518450 7194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.61" port: 40505 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:04:33 dist-test-slave-n106 krb5kdc[7034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265873, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for kudu/127.25.109.61@KRBTEST.COM
Sep 24 16:04:33 dist-test-slave-n106 krb5kdc[7034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265873, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.61@KRBTEST.COM for kudu/127.25.109.61@KRBTEST.COM
I20260924 16:04:33.525321 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.513328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:40505 (local address 127.0.0.1:52568)
0924 16:04:33.513533 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:33.513548 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:33.513620 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:33.514185 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:33.514191 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:33.514404 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:33.514556 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:33.514565 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.516159 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.516163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:33.517106 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:33.517119 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.517789 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.517792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:33.517905 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:33.518372 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:33.518384 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:33.521100 (+ 2716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:33.523335 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:33.523338 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:33.523341 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:33.523573 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:33.523958 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:33.523962 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:33.523964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:33.524061 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:33.524680 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:33.524683 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:33.524841 (+ 158us) client_negotiation.cc:769] Sending connection context
0924 16:04:33.524985 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 16:04:33.525132 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
I20260924 16:04:33.525542 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.513438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52568 (local address 127.25.109.61:40505)
0924 16:04:33.513780 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:33.513791 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:33.513821 (+ 30us) server_negotiation.cc:407] Connection header received
0924 16:04:33.513858 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:33.513861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:33.514029 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:33.514193 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:33.514707 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.515990 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.517247 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.517668 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.517793 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:33.521230 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:33.521240 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:33.521255 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:33.521279 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:33.523211 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:33.523723 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:33.523732 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:33.523737 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:33.523828 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:33.524154 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:33.524157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:33.524159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:33.524535 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:33.524694 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:33.525082 (+ 388us) server_negotiation.cc:299] Negotiation successful
0924 16:04:33.525252 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":163,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.60.keytab.
Entry for principal kudu/127.25.109.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.60.keytab.
I20260924 16:04:33.528043 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.519001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:40505 (local address 127.0.0.1:52582)
0924 16:04:33.519110 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:33.519124 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:33.519198 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:33.519565 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:33.519567 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:33.519744 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:33.519831 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:33.519836 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.520937 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.520940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:33.521633 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:33.521642 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.522165 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.522167 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:33.522228 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:33.522647 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:33.522658 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:33.523824 (+ 1166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:33.526384 (+ 2560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:33.526390 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:33.526393 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:33.526614 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:33.526943 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:33.526949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:33.526951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:33.527043 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:33.527474 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:33.527478 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:33.527578 (+ 100us) client_negotiation.cc:769] Sending connection context
0924 16:04:33.527693 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 16:04:33.527832 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 16:04:33.528055 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.519077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52582 (local address 127.25.109.61:40505)
0924 16:04:33.519354 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:33.519361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:33.519372 (+ 11us) server_negotiation.cc:407] Connection header received
0924 16:04:33.519400 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:33.519402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:33.519434 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:33.519542 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:33.519937 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.520820 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.521743 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.521971 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.522085 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:33.523970 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:33.523973 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:33.523975 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:33.523994 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:33.526280 (+ 2286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:33.526708 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:33.526711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:33.526713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:33.526801 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:33.527159 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:33.527162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:33.527163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:33.527379 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:33.527472 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:33.527751 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 16:04:33.527900 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":218,"thread_start_us":109,"threads_started":1}
I20260924 16:04:33.528093 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.60:38417
--webserver_interface=127.25.109.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.60.keytab
--principal=kudu/127.25.109.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:40627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:41325,127.25.109.61:40505,127.25.109.60:38417
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:04:33.529489 7117 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.60" port: 38417 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:04:33.529855 7194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.60" port: 38417 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:04:33.530324 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.529981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:38417 (local address 127.0.0.1:44534)
0924 16:04:33.530193 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:33.530248 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260924 16:04:33.530573 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.530288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:38417 (local address 127.0.0.1:44542)
0924 16:04:33.530434 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:33.530508 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260924 16:04:33.530761 7143 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.60:38417: connect: Connection refused (error 111)
I20260924 16:04:33.531180 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.530890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:38417 (local address 127.0.0.1:44548)
0924 16:04:33.531067 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:33.531113 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260924 16:04:33.531311 7117 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.109.60:38417: Network error: Client connection negotiation failed: client connection to 127.25.109.60:38417: connect: Connection refused (error 111)
I20260924 16:04:33.531741 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.531535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:38417 (local address 127.0.0.1:44562)
0924 16:04:33.531645 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:33.531687 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 16:04:33.531867 7194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.109.60:38417: Network error: Client connection negotiation failed: client connection to 127.25.109.60:38417: connect: Connection refused (error 111)
I20260924 16:04:33.562476 7117 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.109.60" port: 38417 } attempt: 1
I20260924 16:04:33.563426 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.563062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:38417 (local address 127.0.0.1:44576)
0924 16:04:33.563257 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:33.563331 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260924 16:04:33.564239 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.564016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:38417 (local address 127.0.0.1:44584)
0924 16:04:33.564131 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:33.564185 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 16:04:33.564435 7117 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.109.60:38417: Network error: Client connection negotiation failed: client connection to 127.25.109.60:38417: connect: Connection refused (error 111)
I20260924 16:04:33.581162 7194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.109.60" port: 38417 } attempt: 1
I20260924 16:04:33.582013 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.581722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:38417 (local address 127.0.0.1:44600)
0924 16:04:33.581869 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:33.581939 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 16:04:33.582835 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.582585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:38417 (local address 127.0.0.1:44616)
0924 16:04:33.582712 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:33.582769 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 16:04:33.582971 7194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.109.60:38417: Network error: Client connection negotiation failed: client connection to 127.25.109.60:38417: connect: Connection refused (error 111)
I20260924 16:04:33.639604 7117 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.109.60" port: 38417 } attempt: 2
I20260924 16:04:33.640484 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.640158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:38417 (local address 127.0.0.1:44628)
0924 16:04:33.640328 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:33.640392 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260924 16:04:33.640650 7201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:33.641007 7201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:33.641078 7201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 16:04:33.641275 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.641001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:38417 (local address 127.0.0.1:44630)
0924 16:04:33.641163 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:33.641216 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260924 16:04:33.641484 7117 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.109.60:38417: Network error: Client connection negotiation failed: client connection to 127.25.109.60:38417: connect: Connection refused (error 111)
W20260924 16:04:33.645278 7201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:04:33.645349 7201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:33.645395 7201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:33.645430 7201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:04:33.645465 7201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:04:33.650681 7201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:40627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:41325,127.25.109.61:40505,127.25.109.60:38417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.60.keytab
--principal=kudu/127.25.109.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.60:38417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.109.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:04:33.651121 7194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.109.60" port: 38417 } attempt: 2
I20260924 16:04:33.651914 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.651603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:38417 (local address 127.0.0.1:44632)
0924 16:04:33.651764 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:33.651831 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 16:04:33.652024 7201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:33.652711 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.652494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:38417 (local address 127.0.0.1:44644)
0924 16:04:33.652602 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:33.652653 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 16:04:33.652886 7194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.109.60:38417: Network error: Client connection negotiation failed: client connection to 127.25.109.60:38417: connect: Connection refused (error 111)
I20260924 16:04:33.653347 7201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:33.661810 7208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:33.661832 7210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:33.661832 7207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:33.662138 7201 server_base.cc:1034] running on GCE node
I20260924 16:04:33.662762 7201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:33.663936 7201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:33.665144 7201 hybrid_clock.cc:648] HybridClock initialized: now 1790265873665127 us; error 28 us; skew 500 ppm
Sep 24 16:04:33 dist-test-slave-n106 krb5kdc[7034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265873, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:33.667871 7201 init.cc:377] Logged in from keytab as kudu/127.25.109.60@KRBTEST.COM (short username kudu)
I20260924 16:04:33.668836 7201 webserver.cc:460] Webserver started at http://127.25.109.60:40899/ using document root <none> and password file <none>
I20260924 16:04:33.669353 7201 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:33.669407 7201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:33.669581 7201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:33.671561 7201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-2/data/instance:
uuid: "7bebf3e817d543c59fb4ed0af29eda40"
format_stamp: "Formatted at 2026-09-24 16:04:33 on dist-test-slave-n106"
server_key: "90df9c66a5d505821998eec7de19c93e"
server_key_iv: "1052a29d70f5503926912adbddbb203f"
server_key_version: "encryptionkey@0"
I20260924 16:04:33.672065 7201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-2/wal/instance:
uuid: "7bebf3e817d543c59fb4ed0af29eda40"
format_stamp: "Formatted at 2026-09-24 16:04:33 on dist-test-slave-n106"
server_key: "90df9c66a5d505821998eec7de19c93e"
server_key_iv: "1052a29d70f5503926912adbddbb203f"
server_key_version: "encryptionkey@0"
I20260924 16:04:33.675766 7201 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 16:04:33.678188 7217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:33.679426 7201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 16:04:33.679579 7201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-2/wal
uuid: "7bebf3e817d543c59fb4ed0af29eda40"
format_stamp: "Formatted at 2026-09-24 16:04:33 on dist-test-slave-n106"
server_key: "90df9c66a5d505821998eec7de19c93e"
server_key_iv: "1052a29d70f5503926912adbddbb203f"
server_key_version: "encryptionkey@0"
I20260924 16:04:33.679692 7201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:33.699664 7201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:33.702685 7201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:33.702898 7201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:33.710178 7201 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.60:38417
I20260924 16:04:33.710199 7271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.60:38417 every 8 connection(s)
I20260924 16:04:33.711179 7201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-2/data/info.pb
I20260924 16:04:33.713928 7272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:33.714728 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 7201
I20260924 16:04:33.714855 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265708917108-26036-0/minicluster-data/master-2/wal/instance
I20260924 16:04:33.715188 26036 external_mini_cluster.cc:1428] Setting key baf5b64c8fff2fa833b2c4edf433e314
I20260924 16:04:33.718312 7272 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41325 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:04:33 dist-test-slave-n106 krb5kdc[7034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.60@KRBTEST.COM
Sep 24 16:04:33 dist-test-slave-n106 krb5kdc[7034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265873, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.60@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:04:33.729230 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.719974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37118 (local address 127.25.109.62:41325)
0924 16:04:33.720121 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:33.720126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:33.720391 (+ 265us) server_negotiation.cc:407] Connection header received
0924 16:04:33.720582 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:33.720585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:33.720647 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:33.720742 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:33.721335 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.721855 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.722778 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.723157 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.723286 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:33.725556 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:33.725560 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:33.725563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:33.725594 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:33.727300 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:33.727792 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:33.727794 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:33.727796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:33.727865 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:33.728134 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:33.728136 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:33.728137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:33.728348 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:33.728471 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:33.728925 (+ 454us) server_negotiation.cc:299] Negotiation successful
0924 16:04:33.729033 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 16:04:33.730008 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.716641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44650 (local address 127.25.109.60:38417)
0924 16:04:33.717080 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:33.717095 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:33.717132 (+ 37us) server_negotiation.cc:407] Connection header received
0924 16:04:33.717762 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:33.717779 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:33.718117 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:33.718452 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:33.719266 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.720499 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.721422 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.721880 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.722066 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:33.724973 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:33.724979 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:33.724989 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:33.725010 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:33.726738 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:33.727229 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:33.727234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:33.727239 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:33.727378 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:33.727657 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:33.727660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:33.727662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:33.728212 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:33.728348 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:33.728679 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 16:04:33.728972 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"spinlock_wait_cycles":15104,"thread_start_us":126,"threads_started":1}
I20260924 16:04:33.730008 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.719900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:41325 (local address 127.0.0.1:37118)
0924 16:04:33.720262 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:33.720309 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:33.720463 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:33.720788 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:33.720792 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:33.721097 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:33.721210 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:33.721216 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.721973 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.721978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:33.722659 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:33.722669 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.723300 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.723303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:33.723378 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:33.723926 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:33.723945 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:33.725421 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:33.727404 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:33.727415 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:33.727423 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:33.727694 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:33.727955 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:33.727958 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:33.727959 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:33.728043 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:33.728490 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:33.728495 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:33.728656 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 16:04:33.729375 (+ 719us) client_negotiation.cc:240] Negotiation successful
0924 16:04:33.729541 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":124,"threads_started":1}
I20260924 16:04:33.731298 7272 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.61" port: 40505 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:04:33 dist-test-slave-n106 krb5kdc[7034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265873, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.60@KRBTEST.COM for kudu/127.25.109.61@KRBTEST.COM
Sep 24 16:04:33 dist-test-slave-n106 krb5kdc[7034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:04:33.740008 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.731772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:40505 (local address 127.0.0.1:52588)
0924 16:04:33.731964 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:33.731977 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:33.732049 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:33.732351 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:33.732355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:33.732547 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:33.732644 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:33.732653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.733525 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.733532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:33.734104 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:33.734114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.734676 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.734678 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:33.734739 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:33.735203 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:33.735216 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:33.736660 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:33.738475 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:33.738478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:33.738480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:33.738731 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:33.739033 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:33.739036 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:33.739038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:33.739141 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:33.739513 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:33.739519 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:33.739635 (+ 116us) client_negotiation.cc:769] Sending connection context
0924 16:04:33.739757 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 16:04:33.739891 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260924 16:04:33.740197 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.731894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52588 (local address 127.25.109.61:40505)
0924 16:04:33.732028 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:33.732033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:33.732048 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:04:33.732159 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:33.732162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:33.732218 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:33.732303 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:33.732766 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.733407 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.734210 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.734565 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.734681 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:33.736805 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:33.736809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:33.736812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:33.736844 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:33.738363 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:33.738831 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:33.738836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:33.738840 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:33.738931 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:33.739241 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:33.739243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:33.739244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:33.739410 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:33.739486 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:33.739887 (+ 401us) server_negotiation.cc:299] Negotiation successful
0924 16:04:33.740010 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:04:33.740622 7117 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.109.60" port: 38417 } attempt: 3
I20260924 16:04:33.741101 7272 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.60" port: 38417 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:04:33.742200 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.732578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37122 (local address 127.25.109.62:41325)
0924 16:04:33.732681 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:33.732685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:33.732844 (+ 159us) server_negotiation.cc:407] Connection header received
0924 16:04:33.732880 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:33.732882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:33.732919 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:33.732990 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:33.733574 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.734043 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.735010 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.735218 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.735319 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:33.737968 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:33.737973 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:33.737981 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:33.738003 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:33.739712 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:33.740192 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:33.740195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:33.740196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:33.740285 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:33.740633 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:33.740640 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:33.740644 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:33.740891 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:33.741154 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:33.741596 (+ 442us) server_negotiation.cc:299] Negotiation successful
0924 16:04:33.742056 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 16:04:33.744038 7194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.109.60" port: 38417 } attempt: 3
Sep 24 16:04:33 dist-test-slave-n106 krb5kdc[7034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265873, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for kudu/127.25.109.60@KRBTEST.COM
Sep 24 16:04:33 dist-test-slave-n106 krb5kdc[7034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265873, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.60@KRBTEST.COM for kudu/127.25.109.60@KRBTEST.COM
Sep 24 16:04:33 dist-test-slave-n106 krb5kdc[7034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.61@KRBTEST.COM
I20260924 16:04:33.751312 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.741084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:38417 (local address 127.0.0.1:44652)
0924 16:04:33.741181 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:33.741196 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:33.741271 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:33.741622 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:33.741625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:33.741832 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:33.741908 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:33.741913 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.743443 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.743446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:33.744155 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:33.744165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.745306 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.745310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:33.745397 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:33.745862 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:33.745877 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:33.747338 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:33.749524 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:33.749530 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:33.749535 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:33.749788 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:33.750101 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:33.750104 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:33.750106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:33.750224 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:33.750746 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:33.750752 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:33.750851 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 16:04:33.750976 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 16:04:33.751118 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 16:04:33.752182 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.741275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44652 (local address 127.25.109.60:38417)
0924 16:04:33.741378 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:33.741382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:33.741393 (+ 11us) server_negotiation.cc:407] Connection header received
0924 16:04:33.741451 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:33.741455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:33.741502 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:33.741722 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:33.742019 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.743297 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.744796 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.745189 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.745286 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:33.747510 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:33.747516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:33.747522 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:33.747551 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:33.749348 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:33.749897 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:33.749901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:33.749902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:33.749982 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:33.750334 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:33.750339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:33.750341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:33.750634 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:33.750735 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:33.751278 (+ 543us) server_negotiation.cc:299] Negotiation successful
0924 16:04:33.751961 (+ 683us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":45}
I20260924 16:04:33.752610 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.741568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:38417 (local address 127.0.0.1:44660)
0924 16:04:33.742481 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:33.742496 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:33.742573 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:33.742792 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:33.742795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:33.742950 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:33.743006 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:33.743011 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.744228 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.744235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:33.745423 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:33.745472 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.746191 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.746195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:33.746334 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:33.746800 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:33.746815 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:33.748211 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:33.750228 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:33.750232 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:33.750235 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:33.750466 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:33.750766 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:33.750770 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:33.750773 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:33.750860 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:33.751428 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:33.751432 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:33.751527 (+ 95us) client_negotiation.cc:769] Sending connection context
0924 16:04:33.751631 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 16:04:33.752051 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"mutex_wait_us":343}
I20260924 16:04:33.752391 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.742132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44660 (local address 127.25.109.60:38417)
0924 16:04:33.742527 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:33.742537 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:33.742550 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:04:33.742646 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:33.742650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:33.742682 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:33.742781 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:33.743131 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.743900 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.745583 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.745976 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.746082 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:33.748338 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:33.748342 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:33.748347 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:33.748374 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:33.750120 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:33.750563 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:33.750566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:33.750570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:33.750667 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:33.750948 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:33.750952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:33.750953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:33.751112 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:33.751231 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:33.751680 (+ 449us) server_negotiation.cc:299] Negotiation successful
0924 16:04:33.752158 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":378,"server-negotiator.queue_time_us":336,"thread_start_us":111,"threads_started":1}
Sep 24 16:04:33 dist-test-slave-n106 krb5kdc[7034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265873, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.61@KRBTEST.COM for kudu/127.25.109.60@KRBTEST.COM
I20260924 16:04:33.754014 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.743392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52604 (local address 127.25.109.61:40505)
0924 16:04:33.743518 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:33.743521 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:33.743533 (+ 12us) server_negotiation.cc:407] Connection header received
0924 16:04:33.743564 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:33.743566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:33.743603 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:33.743688 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:33.744466 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.745095 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.746251 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.746609 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.746715 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:33.749792 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:33.749796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:33.749798 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:33.749825 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:33.752248 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:33.752788 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:33.752793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:33.752796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:33.752890 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:33.753192 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:33.753195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:33.753197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:33.753414 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:33.753635 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:33.753755 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 16:04:33.753865 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 16:04:33.756489 7117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9: Bootstrap starting.
I20260924 16:04:33.756488 7272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bebf3e817d543c59fb4ed0af29eda40: Bootstrap starting.
I20260924 16:04:33.759380 7117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:33.759380 7272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bebf3e817d543c59fb4ed0af29eda40: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:33.759541 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.744441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:38417 (local address 127.0.0.1:44676)
0924 16:04:33.744596 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:33.744608 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:33.744676 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:33.745416 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:33.745419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:33.745904 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:33.746007 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:33.746018 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.748075 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.748082 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:33.749790 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:33.749803 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.750932 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.750938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:33.751025 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:33.751649 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:33.751667 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:33.753132 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:33.758096 (+ 4964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:33.758100 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:33.758102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:33.758299 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:33.758583 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:33.758586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:33.758588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:33.758675 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:33.759043 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:33.759046 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:33.759152 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 16:04:33.759252 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 16:04:33.759384 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 16:04:33.759598 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.744513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44676 (local address 127.25.109.60:38417)
0924 16:04:33.745183 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:33.745191 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:33.745203 (+ 12us) server_negotiation.cc:407] Connection header received
0924 16:04:33.745235 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:33.745239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:33.745285 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:33.746119 (+ 834us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:33.746297 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.747707 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.749949 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.750599 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.750925 (+ 326us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:33.753278 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:33.753281 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:33.753284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:33.753308 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:33.757952 (+ 4644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:33.758406 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:33.758409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:33.758411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:33.758489 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:33.758772 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:33.758775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:33.758777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:33.758945 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:33.759050 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:33.759319 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 16:04:33.759424 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":599,"thread_start_us":121,"threads_started":1}
I20260924 16:04:33.760288 7272 log.cc:824] T 00000000000000000000000000000000 P 7bebf3e817d543c59fb4ed0af29eda40: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:33.760288 7117 log.cc:824] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:33.762498 7117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9: No bootstrap required, opened a new log
I20260924 16:04:33.762566 7194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31bd609145e04f2eb4d057535e074336: Bootstrap starting.
I20260924 16:04:33.762679 7272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bebf3e817d543c59fb4ed0af29eda40: No bootstrap required, opened a new log
I20260924 16:04:33.763880 26036 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:04:33.765234 7194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31bd609145e04f2eb4d057535e074336: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:33.766362 7194 log.cc:824] T 00000000000000000000000000000000 P 31bd609145e04f2eb4d057535e074336: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:33.767066 7117 raft_consensus.cc:348] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70f5176f8a64d50a5534b205a949fd9" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41325 } } peers { permanent_uuid: "31bd609145e04f2eb4d057535e074336" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 40505 } } peers { permanent_uuid: "7bebf3e817d543c59fb4ed0af29eda40" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 38417 } }
I20260924 16:04:33.767400 7117 raft_consensus.cc:374] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:33.767493 7117 raft_consensus.cc:729] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e70f5176f8a64d50a5534b205a949fd9, State: Initialized, Role: FOLLOWER
I20260924 16:04:33.767472 7272 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bebf3e817d543c59fb4ed0af29eda40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70f5176f8a64d50a5534b205a949fd9" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41325 } } peers { permanent_uuid: "31bd609145e04f2eb4d057535e074336" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 40505 } } peers { permanent_uuid: "7bebf3e817d543c59fb4ed0af29eda40" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 38417 } }
I20260924 16:04:33.767710 7272 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bebf3e817d543c59fb4ed0af29eda40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:33.767781 7272 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bebf3e817d543c59fb4ed0af29eda40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bebf3e817d543c59fb4ed0af29eda40, State: Initialized, Role: FOLLOWER
I20260924 16:04:33.768184 7117 consensus_queue.cc:260] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70f5176f8a64d50a5534b205a949fd9" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41325 } } peers { permanent_uuid: "31bd609145e04f2eb4d057535e074336" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 40505 } } peers { permanent_uuid: "7bebf3e817d543c59fb4ed0af29eda40" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 38417 } }
I20260924 16:04:33.768270 7272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bebf3e817d543c59fb4ed0af29eda40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70f5176f8a64d50a5534b205a949fd9" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41325 } } peers { permanent_uuid: "31bd609145e04f2eb4d057535e074336" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 40505 } } peers { permanent_uuid: "7bebf3e817d543c59fb4ed0af29eda40" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 38417 } }
I20260924 16:04:33.768613 7194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31bd609145e04f2eb4d057535e074336: No bootstrap required, opened a new log
I20260924 16:04:33.769647 7272 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bebf3e817d543c59fb4ed0af29eda40 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:04:33.769647 7117 sys_catalog.cc:564] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:04:33.772751 7194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31bd609145e04f2eb4d057535e074336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70f5176f8a64d50a5534b205a949fd9" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41325 } } peers { permanent_uuid: "31bd609145e04f2eb4d057535e074336" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 40505 } } peers { permanent_uuid: "7bebf3e817d543c59fb4ed0af29eda40" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 38417 } }
I20260924 16:04:33.772981 7194 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31bd609145e04f2eb4d057535e074336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:33.773033 7194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31bd609145e04f2eb4d057535e074336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31bd609145e04f2eb4d057535e074336, State: Initialized, Role: FOLLOWER
I20260924 16:04:33.773502 7194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31bd609145e04f2eb4d057535e074336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70f5176f8a64d50a5534b205a949fd9" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41325 } } peers { permanent_uuid: "31bd609145e04f2eb4d057535e074336" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 40505 } } peers { permanent_uuid: "7bebf3e817d543c59fb4ed0af29eda40" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 38417 } }
I20260924 16:04:33.774880 7194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31bd609145e04f2eb4d057535e074336 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 16:04:33.779608 7313 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7bebf3e817d543c59fb4ed0af29eda40: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:04:33.779688 7313 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 16:04:33.780277 7326 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 31bd609145e04f2eb4d057535e074336: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:04:33.780449 7326 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:04:33.776938 7047 master_runner.cc:186] Error getting master registration for 127.25.109.61:40505: OK, instance_id { permanent_uuid: "31bd609145e04f2eb4d057535e074336" instance_seqno: 1790265873490770 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 16:04:33.781001 7047 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 16:04:33.781001 7290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bebf3e817d543c59fb4ed0af29eda40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70f5176f8a64d50a5534b205a949fd9" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41325 } } peers { permanent_uuid: "31bd609145e04f2eb4d057535e074336" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 40505 } } peers { permanent_uuid: "7bebf3e817d543c59fb4ed0af29eda40" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 38417 } } }
I20260924 16:04:33.781133 7290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bebf3e817d543c59fb4ed0af29eda40 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:04:33.781232 7315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31bd609145e04f2eb4d057535e074336 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70f5176f8a64d50a5534b205a949fd9" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41325 } } peers { permanent_uuid: "31bd609145e04f2eb4d057535e074336" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 40505 } } peers { permanent_uuid: "7bebf3e817d543c59fb4ed0af29eda40" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 38417 } } }
I20260924 16:04:33.781353 7315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31bd609145e04f2eb4d057535e074336 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:04:33.782575 7291 sys_catalog.cc:455] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70f5176f8a64d50a5534b205a949fd9" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41325 } } peers { permanent_uuid: "31bd609145e04f2eb4d057535e074336" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 40505 } } peers { permanent_uuid: "7bebf3e817d543c59fb4ed0af29eda40" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 38417 } } }
I20260924 16:04:33.782702 7291 sys_catalog.cc:458] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:04:33.784464 7124 master_runner.cc:263] No leader master found from master 31bd609145e04f2eb4d057535e074336
I20260924 16:04:33.784480 7201 master_runner.cc:263] No leader master found from master 7bebf3e817d543c59fb4ed0af29eda40
I20260924 16:04:33.784533 7201 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 16:04:33.784541 7124 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 16:04:33.785495 7314 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:04:33.785561 7314 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:04:33.785944 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.766202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44678 (local address 127.25.109.60:38417)
0924 16:04:33.766320 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:33.766324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:33.768253 (+ 1929us) server_negotiation.cc:407] Connection header received
0924 16:04:33.768459 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:33.768462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:33.768510 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:33.768568 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:33.769523 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.770310 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.771573 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.771938 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.772014 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:33.773910 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:33.773914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:33.773917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:33.773942 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:33.783271 (+ 9329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:33.783863 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:33.783868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:33.783872 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:33.784013 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:33.784861 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:33.784865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:33.784867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:33.785073 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:33.785196 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:33.785642 (+ 446us) server_negotiation.cc:299] Negotiation successful
0924 16:04:33.785760 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 16:04:33.787954 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.766913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37126 (local address 127.25.109.62:41325)
0924 16:04:33.767051 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:33.767055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:33.771835 (+ 4780us) server_negotiation.cc:407] Connection header received
0924 16:04:33.772398 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:33.772428 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:33.772479 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:33.772538 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:33.773502 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.774284 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.775412 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.775703 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.775781 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:33.781295 (+ 5514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:33.781299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:33.781302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:33.781330 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:33.784225 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:33.785099 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:33.785103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:33.785105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:33.785193 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:33.785553 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:33.785558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:33.785559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:33.785792 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:33.785903 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:33.787693 (+ 1790us) server_negotiation.cc:299] Negotiation successful
0924 16:04:33.787812 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:04:33.791388 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:33.767536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52608 (local address 127.25.109.61:40505)
0924 16:04:33.767654 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:33.767658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:33.781548 (+ 13890us) server_negotiation.cc:407] Connection header received
0924 16:04:33.781713 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:33.781717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:33.781773 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:33.781866 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:33.783966 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.784694 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.785901 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:33.786222 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:33.786313 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:33.787905 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:33.787909 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:33.787913 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:33.787939 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:33.789589 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:33.790072 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:33.790076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:33.790077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:33.790175 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:33.790492 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:33.790497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:33.790499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:33.790713 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:33.790840 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:33.791112 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 16:04:33.791238 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 16:04:33.827468 7291 raft_consensus.cc:482] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:04:33.827688 7291 raft_consensus.cc:504] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70f5176f8a64d50a5534b205a949fd9" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41325 } } peers { permanent_uuid: "31bd609145e04f2eb4d057535e074336" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 40505 } } peers { permanent_uuid: "7bebf3e817d543c59fb4ed0af29eda40" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 38417 } }
I20260924 16:04:33.828998 7291 leader_election.cc:288] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31bd609145e04f2eb4d057535e074336 (127.25.109.61:40505), 7bebf3e817d543c59fb4ed0af29eda40 (127.25.109.60:38417)
I20260924 16:04:33.829902 7165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e70f5176f8a64d50a5534b205a949fd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31bd609145e04f2eb4d057535e074336" is_pre_election: true
I20260924 16:04:33.830200 7165 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 31bd609145e04f2eb4d057535e074336 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e70f5176f8a64d50a5534b205a949fd9 in term 0.
I20260924 16:04:33.830611 7245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e70f5176f8a64d50a5534b205a949fd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bebf3e817d543c59fb4ed0af29eda40" is_pre_election: true
I20260924 16:04:33.830878 7245 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7bebf3e817d543c59fb4ed0af29eda40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e70f5176f8a64d50a5534b205a949fd9 in term 0.
I20260924 16:04:33.831344 7063 leader_election.cc:302] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31bd609145e04f2eb4d057535e074336, e70f5176f8a64d50a5534b205a949fd9; no voters:
I20260924 16:04:33.832140 7291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:04:33.832224 7291 raft_consensus.cc:482] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:04:33.832264 7291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:33.833640 7291 raft_consensus.cc:504] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70f5176f8a64d50a5534b205a949fd9" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41325 } } peers { permanent_uuid: "31bd609145e04f2eb4d057535e074336" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 40505 } } peers { permanent_uuid: "7bebf3e817d543c59fb4ed0af29eda40" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 38417 } }
I20260924 16:04:33.834604 7245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e70f5176f8a64d50a5534b205a949fd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bebf3e817d543c59fb4ed0af29eda40"
I20260924 16:04:33.834719 7245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bebf3e817d543c59fb4ed0af29eda40 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:33.835980 7245 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7bebf3e817d543c59fb4ed0af29eda40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e70f5176f8a64d50a5534b205a949fd9 in term 1.
I20260924 16:04:33.836354 7066 leader_election.cc:302] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bebf3e817d543c59fb4ed0af29eda40, e70f5176f8a64d50a5534b205a949fd9; no voters:
I20260924 16:04:33.836771 7165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e70f5176f8a64d50a5534b205a949fd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31bd609145e04f2eb4d057535e074336"
I20260924 16:04:33.836894 7165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31bd609145e04f2eb4d057535e074336 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:33.838109 7165 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 31bd609145e04f2eb4d057535e074336 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e70f5176f8a64d50a5534b205a949fd9 in term 1.
I20260924 16:04:33.840948 7291 leader_election.cc:288] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9 [CANDIDATE]: Term 1 election: Requested vote from peers 31bd609145e04f2eb4d057535e074336 (127.25.109.61:40505), 7bebf3e817d543c59fb4ed0af29eda40 (127.25.109.60:38417)
I20260924 16:04:33.841152 7291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:33.841271 7291 raft_consensus.cc:686] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9 [term 1 LEADER]: Becoming Leader. State: Replica: e70f5176f8a64d50a5534b205a949fd9, State: Running, Role: LEADER
I20260924 16:04:33.841658 7291 consensus_queue.cc:237] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70f5176f8a64d50a5534b205a949fd9" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41325 } } peers { permanent_uuid: "31bd609145e04f2eb4d057535e074336" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 40505 } } peers { permanent_uuid: "7bebf3e817d543c59fb4ed0af29eda40" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 38417 } }
I20260924 16:04:33.843573 7291 sys_catalog.cc:455] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e70f5176f8a64d50a5534b205a949fd9. Latest consensus state: current_term: 1 leader_uuid: "e70f5176f8a64d50a5534b205a949fd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70f5176f8a64d50a5534b205a949fd9" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41325 } } peers { permanent_uuid: "31bd609145e04f2eb4d057535e074336" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 40505 } } peers { permanent_uuid: "7bebf3e817d543c59fb4ed0af29eda40" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 38417 } } }
I20260924 16:04:33.843747 7291 sys_catalog.cc:458] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:33.844198 7332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:04:33.845211 7332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:04:33.854493 7245 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7bebf3e817d543c59fb4ed0af29eda40 [term 1 FOLLOWER]: Refusing update from remote peer e70f5176f8a64d50a5534b205a949fd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:04:33.855286 7330 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bebf3e817d543c59fb4ed0af29eda40" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 38417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:04:33.859802 7290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bebf3e817d543c59fb4ed0af29eda40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e70f5176f8a64d50a5534b205a949fd9. Latest consensus state: current_term: 1 leader_uuid: "e70f5176f8a64d50a5534b205a949fd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70f5176f8a64d50a5534b205a949fd9" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41325 } } peers { permanent_uuid: "31bd609145e04f2eb4d057535e074336" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 40505 } } peers { permanent_uuid: "7bebf3e817d543c59fb4ed0af29eda40" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 38417 } } }
I20260924 16:04:33.859923 7290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bebf3e817d543c59fb4ed0af29eda40 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:04:33.860980 7165 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 31bd609145e04f2eb4d057535e074336 [term 1 FOLLOWER]: Refusing update from remote peer e70f5176f8a64d50a5534b205a949fd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:04:33.861649 7330 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31bd609145e04f2eb4d057535e074336" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 40505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:04:33.865283 7315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31bd609145e04f2eb4d057535e074336 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e70f5176f8a64d50a5534b205a949fd9. Latest consensus state: current_term: 1 leader_uuid: "e70f5176f8a64d50a5534b205a949fd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70f5176f8a64d50a5534b205a949fd9" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41325 } } peers { permanent_uuid: "31bd609145e04f2eb4d057535e074336" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 40505 } } peers { permanent_uuid: "7bebf3e817d543c59fb4ed0af29eda40" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 38417 } } }
I20260924 16:04:33.865394 7315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31bd609145e04f2eb4d057535e074336 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:04:33.872884 7330 sys_catalog.cc:455] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e70f5176f8a64d50a5534b205a949fd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70f5176f8a64d50a5534b205a949fd9" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41325 } } peers { permanent_uuid: "31bd609145e04f2eb4d057535e074336" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 40505 } } peers { permanent_uuid: "7bebf3e817d543c59fb4ed0af29eda40" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 38417 } } }
I20260924 16:04:33.873008 7330 sys_catalog.cc:458] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:33.874261 7315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31bd609145e04f2eb4d057535e074336 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e70f5176f8a64d50a5534b205a949fd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70f5176f8a64d50a5534b205a949fd9" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41325 } } peers { permanent_uuid: "31bd609145e04f2eb4d057535e074336" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 40505 } } peers { permanent_uuid: "7bebf3e817d543c59fb4ed0af29eda40" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 38417 } } }
I20260924 16:04:33.874372 7315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31bd609145e04f2eb4d057535e074336 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:04:33.874804 7333 mvcc.cc:205] Tried to move back new op lower bound from 7332929019300528128 to 7332929019258945536. Current Snapshot: MvccSnapshot[applied={T|T < 7332929019300528128}]
I20260924 16:04:33.875229 7336 mvcc.cc:205] Tried to move back new op lower bound from 7332929019300528128 to 7332929019258945536. Current Snapshot: MvccSnapshot[applied={T|T < 7332929019300528128}]
I20260924 16:04:33.877324 7291 sys_catalog.cc:455] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e70f5176f8a64d50a5534b205a949fd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70f5176f8a64d50a5534b205a949fd9" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41325 } } peers { permanent_uuid: "31bd609145e04f2eb4d057535e074336" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 40505 } } peers { permanent_uuid: "7bebf3e817d543c59fb4ed0af29eda40" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 38417 } } }
I20260924 16:04:33.877439 7291 sys_catalog.cc:458] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:33.878409 7290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bebf3e817d543c59fb4ed0af29eda40 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e70f5176f8a64d50a5534b205a949fd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70f5176f8a64d50a5534b205a949fd9" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41325 } } peers { permanent_uuid: "31bd609145e04f2eb4d057535e074336" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 40505 } } peers { permanent_uuid: "7bebf3e817d543c59fb4ed0af29eda40" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 38417 } } }
I20260924 16:04:33.878520 7290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bebf3e817d543c59fb4ed0af29eda40 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:04:33.886919 7332 catalog_manager.cc:1347] Generated new cluster ID: 328506d487be4786b8e43a4f09ff5eb8
I20260924 16:04:33.886988 7332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:04:33.902336 7334 mvcc.cc:205] Tried to move back new op lower bound from 7332929019300528128 to 7332929019258945536. Current Snapshot: MvccSnapshot[applied={T|T < 7332929019300528128}]
I20260924 16:04:33.914402 7332 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:04:33.915850 7332 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:04:33.936244 7332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e70f5176f8a64d50a5534b205a949fd9: Generated new TSK 0
I20260924 16:04:33.939797 7332 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:04:34.781019 7313 catalog_manager.cc:1259] Loaded cluster ID: 328506d487be4786b8e43a4f09ff5eb8
I20260924 16:04:34.781093 7313 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7bebf3e817d543c59fb4ed0af29eda40: loading cluster ID for follower catalog manager: success
I20260924 16:04:34.781659 7326 catalog_manager.cc:1259] Loaded cluster ID: 328506d487be4786b8e43a4f09ff5eb8
I20260924 16:04:34.781741 7326 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 31bd609145e04f2eb4d057535e074336: loading cluster ID for follower catalog manager: success
I20260924 16:04:34.782800 7313 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7bebf3e817d543c59fb4ed0af29eda40: acquiring CA information for follower catalog manager: success
I20260924 16:04:34.783382 7326 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 31bd609145e04f2eb4d057535e074336: acquiring CA information for follower catalog manager: success
I20260924 16:04:34.784130 7313 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7bebf3e817d543c59fb4ed0af29eda40: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:04:34.784591 7326 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 31bd609145e04f2eb4d057535e074336: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:04:35.015751 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 7047
I20260924 16:04:35.022707 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 7124
I20260924 16:04:35.028654 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 7201
2026-09-24T16:04:35Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3104 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:04:35 dist-test-slave-n106 krb5kdc[7348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:04:35 dist-test-slave-n106 krb5kdc[7348](info): set up 2 sockets
Sep 24 16:04:35 dist-test-slave-n106 krb5kdc[7348](info): commencing operation
krb5kdc: starting...
W20260924 16:04:36.065429 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:04:36 dist-test-slave-n106 krb5kdc[7348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:04:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:04:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
I20260924 16:04:36.185073 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:36965
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:44471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:36965
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:36.293483 7362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:36.293756 7362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:36.293797 7362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:36.297868 7362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:04:36.297924 7362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:36.297943 7362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:36.297960 7362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:04:36.298012 7362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:04:36.303216 7362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:44471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:36965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:36965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:04:36.304337 7362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:36.305402 7362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:36.313714 7370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:36.313712 7367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:36.313861 7362 server_base.cc:1034] running on GCE node
W20260924 16:04:36.314040 7368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:36.314589 7362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:36.315629 7362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:36.316810 7362 hybrid_clock.cc:648] HybridClock initialized: now 1790265876316778 us; error 47 us; skew 500 ppm
Sep 24 16:04:36 dist-test-slave-n106 krb5kdc[7348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265876, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:36.319394 7362 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:04:36.320225 7362 webserver.cc:460] Webserver started at http://127.25.109.62:39599/ using document root <none> and password file <none>
I20260924 16:04:36.320833 7362 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:36.320928 7362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:36.321169 7362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:36.323156 7362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "6b63722c8c5f4332866bc06709f8e9cd"
format_stamp: "Formatted at 2026-09-24 16:04:36 on dist-test-slave-n106"
server_key: "a915a0b98d7b8d192d70245e01cae1d2"
server_key_iv: "6d35283b85cf510f2b67689b08de9271"
server_key_version: "encryptionkey@0"
I20260924 16:04:36.323726 7362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "6b63722c8c5f4332866bc06709f8e9cd"
format_stamp: "Formatted at 2026-09-24 16:04:36 on dist-test-slave-n106"
server_key: "a915a0b98d7b8d192d70245e01cae1d2"
server_key_iv: "6d35283b85cf510f2b67689b08de9271"
server_key_version: "encryptionkey@0"
I20260924 16:04:36.327363 7362 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 16:04:36.329743 7377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:36.331017 7362 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:04:36.331179 7362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "6b63722c8c5f4332866bc06709f8e9cd"
format_stamp: "Formatted at 2026-09-24 16:04:36 on dist-test-slave-n106"
server_key: "a915a0b98d7b8d192d70245e01cae1d2"
server_key_iv: "6d35283b85cf510f2b67689b08de9271"
server_key_version: "encryptionkey@0"
I20260924 16:04:36.331295 7362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:36.343866 7362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:36.346868 7362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:36.347101 7362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:36.354545 7362 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:36965
I20260924 16:04:36.354573 7429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:36965 every 8 connection(s)
I20260924 16:04:36.355811 7362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:04:36.358762 7430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:36.360662 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 7362
I20260924 16:04:36.360795 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:04:36.361120 26036 external_mini_cluster.cc:1428] Setting key 833f8a93a751a733075a0e742be0cbf8
I20260924 16:04:36.365312 7430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b63722c8c5f4332866bc06709f8e9cd: Bootstrap starting.
I20260924 16:04:36.367581 7430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b63722c8c5f4332866bc06709f8e9cd: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:36.368351 7430 log.cc:824] T 00000000000000000000000000000000 P 6b63722c8c5f4332866bc06709f8e9cd: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:04:36 dist-test-slave-n106 krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:04:36.370632 7430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b63722c8c5f4332866bc06709f8e9cd: No bootstrap required, opened a new log
I20260924 16:04:36.374611 7430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b63722c8c5f4332866bc06709f8e9cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b63722c8c5f4332866bc06709f8e9cd" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 36965 } }
I20260924 16:04:36.374897 7430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b63722c8c5f4332866bc06709f8e9cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:36.374985 7430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b63722c8c5f4332866bc06709f8e9cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b63722c8c5f4332866bc06709f8e9cd, State: Initialized, Role: FOLLOWER
I20260924 16:04:36.375487 7430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b63722c8c5f4332866bc06709f8e9cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b63722c8c5f4332866bc06709f8e9cd" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 36965 } }
I20260924 16:04:36.375572 7433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:36.362566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52058 (local address 127.25.109.62:36965)
0924 16:04:36.363031 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:36.363047 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:36.363085 (+ 38us) server_negotiation.cc:407] Connection header received
0924 16:04:36.363764 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:36.363782 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:36.364115 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:36.364478 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:36.365341 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:36.366376 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:36.367258 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:36.367803 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:36.367980 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:36.370720 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:36.370747 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:36.370766 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:36.370799 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:36.372551 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:36.373050 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:36.373055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:36.373060 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:36.373183 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:36.373482 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:36.373486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:36.373488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:36.373896 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:36.374157 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:36.374381 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 16:04:36.374685 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":148,"threads_started":1}
I20260924 16:04:36.375643 7430 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b63722c8c5f4332866bc06709f8e9cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:04:36.375713 7430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b63722c8c5f4332866bc06709f8e9cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:04:36.375837 7430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b63722c8c5f4332866bc06709f8e9cd [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:36.376964 7430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b63722c8c5f4332866bc06709f8e9cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b63722c8c5f4332866bc06709f8e9cd" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 36965 } }
I20260924 16:04:36.377314 7430 leader_election.cc:302] T 00000000000000000000000000000000 P 6b63722c8c5f4332866bc06709f8e9cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b63722c8c5f4332866bc06709f8e9cd; no voters:
I20260924 16:04:36.377687 7430 leader_election.cc:288] T 00000000000000000000000000000000 P 6b63722c8c5f4332866bc06709f8e9cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:04:36.377760 7435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b63722c8c5f4332866bc06709f8e9cd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:36.377952 7435 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b63722c8c5f4332866bc06709f8e9cd [term 1 LEADER]: Becoming Leader. State: Replica: 6b63722c8c5f4332866bc06709f8e9cd, State: Running, Role: LEADER
I20260924 16:04:36.378255 7435 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b63722c8c5f4332866bc06709f8e9cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b63722c8c5f4332866bc06709f8e9cd" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 36965 } }
I20260924 16:04:36.378779 26036 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:04:36.379004 7430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b63722c8c5f4332866bc06709f8e9cd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:04:36.379489 7436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b63722c8c5f4332866bc06709f8e9cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b63722c8c5f4332866bc06709f8e9cd. Latest consensus state: current_term: 1 leader_uuid: "6b63722c8c5f4332866bc06709f8e9cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b63722c8c5f4332866bc06709f8e9cd" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 36965 } } }
I20260924 16:04:36.380283 7436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b63722c8c5f4332866bc06709f8e9cd [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:36.380175 7437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b63722c8c5f4332866bc06709f8e9cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b63722c8c5f4332866bc06709f8e9cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b63722c8c5f4332866bc06709f8e9cd" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 36965 } } }
I20260924 16:04:36.380661 7437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b63722c8c5f4332866bc06709f8e9cd [sys.catalog]: This master's current role is: LEADER
W20260924 16:04:36.383613 7452 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b63722c8c5f4332866bc06709f8e9cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:04:36.383706 7452 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:04:36.383797 7446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:04:36.384653 7446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:04:36.389833 7446 catalog_manager.cc:1347] Generated new cluster ID: c1d26a1e5a00407d8632985d9c7a6947
I20260924 16:04:36.389895 7446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:04:36.408502 7446 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:04:36.409860 7446 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:04:36.425858 7446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b63722c8c5f4332866bc06709f8e9cd: Generated new TSK 0
I20260924 16:04:36.426863 7446 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:04:36.434705 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 7362
2026-09-24T16:04:36Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1405 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:04:36 dist-test-slave-n106 krb5kdc[7459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:04:36 dist-test-slave-n106 krb5kdc[7459](info): set up 2 sockets
Sep 24 16:04:36 dist-test-slave-n106 krb5kdc[7459](info): commencing operation
krb5kdc: starting...
W20260924 16:04:37.469664 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:04:37 dist-test-slave-n106 krb5kdc[7459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:04:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:04:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
I20260924 16:04:37.587833 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:42581
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:41887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:42581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:37.698148 7472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:37.698474 7472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:37.698576 7472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:37.702629 7472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:04:37.702731 7472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:37.702770 7472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:37.702807 7472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:04:37.702837 7472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:04:37.708108 7472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:41887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:42581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:42581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:04:37.709228 7472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:37.710270 7472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:37.718904 7477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:37.718885 7480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:37.718889 7478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:37.719067 7472 server_base.cc:1034] running on GCE node
I20260924 16:04:37.719748 7472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:37.720902 7472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:37.722096 7472 hybrid_clock.cc:648] HybridClock initialized: now 1790265877722062 us; error 45 us; skew 500 ppm
Sep 24 16:04:37 dist-test-slave-n106 krb5kdc[7459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265877, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:37.728572 7472 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:04:37.729513 7472 webserver.cc:460] Webserver started at http://127.25.109.62:38041/ using document root <none> and password file <none>
I20260924 16:04:37.730111 7472 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:37.730201 7472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:37.730443 7472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:37.732601 7472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "38fa2227488d452a92f88ae33c25bf67"
format_stamp: "Formatted at 2026-09-24 16:04:37 on dist-test-slave-n106"
server_key: "17b0a5cdb7457c565552f518246b0f56"
server_key_iv: "43045f6b8bb5329960d699a2ffb73e96"
server_key_version: "encryptionkey@0"
I20260924 16:04:37.733199 7472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "38fa2227488d452a92f88ae33c25bf67"
format_stamp: "Formatted at 2026-09-24 16:04:37 on dist-test-slave-n106"
server_key: "17b0a5cdb7457c565552f518246b0f56"
server_key_iv: "43045f6b8bb5329960d699a2ffb73e96"
server_key_version: "encryptionkey@0"
I20260924 16:04:37.736939 7472 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 16:04:37.739428 7487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:37.740743 7472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:04:37.740892 7472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "38fa2227488d452a92f88ae33c25bf67"
format_stamp: "Formatted at 2026-09-24 16:04:37 on dist-test-slave-n106"
server_key: "17b0a5cdb7457c565552f518246b0f56"
server_key_iv: "43045f6b8bb5329960d699a2ffb73e96"
server_key_version: "encryptionkey@0"
I20260924 16:04:37.741011 7472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:37.777684 7472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:37.781293 7472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:37.781529 7472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:37.788767 7472 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:42581
I20260924 16:04:37.789075 7539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:42581 every 8 connection(s)
I20260924 16:04:37.789831 7472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:04:37.792732 7540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:37.794492 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 7472
I20260924 16:04:37.794646 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:04:37.794973 26036 external_mini_cluster.cc:1428] Setting key 3d9a8fe79d6f567c7f78df320e41257c
I20260924 16:04:37.799481 7540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67: Bootstrap starting.
I20260924 16:04:37.802105 7540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:37.803056 7540 log.cc:824] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:04:37 dist-test-slave-n106 krb5kdc[7459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:04:37.805411 7540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67: No bootstrap required, opened a new log
I20260924 16:04:37.809839 7543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:37.796366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42864 (local address 127.25.109.62:42581)
0924 16:04:37.796868 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:37.796887 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:37.796931 (+ 44us) server_negotiation.cc:407] Connection header received
0924 16:04:37.797558 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:37.797575 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:37.797899 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:37.798184 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:37.799095 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:37.800132 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:37.801311 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:37.801765 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:37.801883 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:37.804694 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:37.804701 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:37.804717 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:37.804750 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:37.806692 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:37.807101 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:37.807107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:37.807114 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:37.807255 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:37.807532 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:37.807535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:37.807537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:37.808110 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:37.808298 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:37.808549 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 16:04:37.808843 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":135,"threads_started":1}
I20260924 16:04:37.810087 7540 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fa2227488d452a92f88ae33c25bf67" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 42581 } }
I20260924 16:04:37.810357 7540 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:37.810415 7540 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38fa2227488d452a92f88ae33c25bf67, State: Initialized, Role: FOLLOWER
I20260924 16:04:37.810869 7540 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fa2227488d452a92f88ae33c25bf67" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 42581 } }
I20260924 16:04:37.811043 7540 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:04:37.811143 7540 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:04:37.811247 7540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:37.812297 7540 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fa2227488d452a92f88ae33c25bf67" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 42581 } }
I20260924 16:04:37.812659 7540 leader_election.cc:302] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38fa2227488d452a92f88ae33c25bf67; no voters:
I20260924 16:04:37.812971 7540 leader_election.cc:288] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:04:37.813107 7545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:37.813336 7545 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [term 1 LEADER]: Becoming Leader. State: Replica: 38fa2227488d452a92f88ae33c25bf67, State: Running, Role: LEADER
I20260924 16:04:37.813690 7545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fa2227488d452a92f88ae33c25bf67" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 42581 } }
I20260924 16:04:37.814280 7540 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:04:37.815369 7547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38fa2227488d452a92f88ae33c25bf67. Latest consensus state: current_term: 1 leader_uuid: "38fa2227488d452a92f88ae33c25bf67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fa2227488d452a92f88ae33c25bf67" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 42581 } } }
I20260924 16:04:37.815506 7547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:37.815804 7546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38fa2227488d452a92f88ae33c25bf67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fa2227488d452a92f88ae33c25bf67" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 42581 } } }
I20260924 16:04:37.815907 7554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:04:37.815905 7546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:37.818689 7554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:04:37.824205 7554 catalog_manager.cc:1347] Generated new cluster ID: d5a4fc66d2284c1299703c4863799ad0
I20260924 16:04:37.824337 7554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:04:37.857201 7554 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:04:37.858386 7554 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:04:37.866946 7554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67: Generated new TSK 0
I20260924 16:04:37.867655 7554 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
I20260924 16:04:37.905134 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:0
--local_ip_for_outbound_sockets=127.25.109.1
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:42581
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:41887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:38.018455 7566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:38.018709 7566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:38.018751 7566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:38.022817 7566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:38.022872 7566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:38.022964 7566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:04:38.028113 7566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:41887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:42581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:04:38.029170 7566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:38.030201 7566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:38.040206 7571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:38.040198 7572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:38.040280 7566 server_base.cc:1034] running on GCE node
W20260924 16:04:38.040198 7574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:38.040956 7566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:38.041536 7566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:38.042698 7566 hybrid_clock.cc:648] HybridClock initialized: now 1790265878042683 us; error 26 us; skew 500 ppm
Sep 24 16:04:38 dist-test-slave-n106 krb5kdc[7459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265878, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:38.045634 7566 init.cc:377] Logged in from keytab as kudu/127.25.109.1@KRBTEST.COM (short username kudu)
I20260924 16:04:38.046422 7566 webserver.cc:460] Webserver started at http://127.25.109.1:44365/ using document root <none> and password file <none>
I20260924 16:04:38.046940 7566 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:38.046998 7566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:38.047158 7566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:38.049207 7566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/data/instance:
uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e"
format_stamp: "Formatted at 2026-09-24 16:04:38 on dist-test-slave-n106"
server_key: "4017e7eca347eb3faa717f9498fc91af"
server_key_iv: "638ec19e00271cde59e6e735835257f0"
server_key_version: "encryptionkey@0"
I20260924 16:04:38.049724 7566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance:
uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e"
format_stamp: "Formatted at 2026-09-24 16:04:38 on dist-test-slave-n106"
server_key: "4017e7eca347eb3faa717f9498fc91af"
server_key_iv: "638ec19e00271cde59e6e735835257f0"
server_key_version: "encryptionkey@0"
I20260924 16:04:38.053443 7566 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:04:38.055800 7581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:38.057075 7566 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:04:38.057212 7566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e"
format_stamp: "Formatted at 2026-09-24 16:04:38 on dist-test-slave-n106"
server_key: "4017e7eca347eb3faa717f9498fc91af"
server_key_iv: "638ec19e00271cde59e6e735835257f0"
server_key_version: "encryptionkey@0"
I20260924 16:04:38.057329 7566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:38.077783 7566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:38.081153 7566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:38.081357 7566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:38.081949 7566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:38.082767 7566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:38.082837 7566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:38.082904 7566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:38.082957 7566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:38.091624 7566 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:33745
I20260924 16:04:38.091673 7699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:33745 every 8 connection(s)
I20260924 16:04:38.092525 7566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:04:38.101846 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 7566
Sep 24 16:04:38 dist-test-slave-n106 krb5kdc[7459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265878, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:04:38.102003 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance
I20260924 16:04:38.102312 26036 external_mini_cluster.cc:1428] Setting key 6a3dcdc6896dc115805b55beb2d6bb85
I20260924 16:04:38.106326 7543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:38.094296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:41425 (local address 127.25.109.62:42581)
0924 16:04:38.094445 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:38.094449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:38.095372 (+ 923us) server_negotiation.cc:407] Connection header received
0924 16:04:38.096809 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:38.096813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:38.096873 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:38.096955 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:38.098109 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.098651 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.099774 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.100017 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.100128 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:38.102605 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:38.102609 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:38.102611 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:38.102651 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:38.104275 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:38.104810 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:38.104816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:38.104818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:38.104901 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:38.105248 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:38.105252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:38.105253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:38.105432 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:38.105541 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:38.106015 (+ 474us) server_negotiation.cc:299] Negotiation successful
0924 16:04:38.106157 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:04:38.107061 7702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:38.094728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.1:41425)
0924 16:04:38.095220 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:38.095263 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:38.096587 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:38.097133 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:38.097141 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:38.097490 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:38.097955 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:38.097967 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.098785 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.098789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:38.099643 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:38.099652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.100188 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.100193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:38.100359 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:38.100982 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:38.100998 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:38.102440 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:38.104394 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:38.104399 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:38.104438 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:38.104705 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:38.105001 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:38.105004 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:38.105006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:38.105147 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:38.105540 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:38.105547 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:38.105779 (+ 232us) client_negotiation.cc:769] Sending connection context
0924 16:04:38.105957 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 16:04:38.106265 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":122,"threads_started":1}
I20260924 16:04:38.108289 7700 heartbeater.cc:347] Connected to a master server at 127.25.109.62:42581
I20260924 16:04:38.108577 7700 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:38.109194 7700 heartbeater.cc:511] Master 127.25.109.62:42581 requested a full tablet report, sending...
I20260924 16:04:38.110792 7504 ts_manager.cc:194] Registered new tserver with Master: ccf0f95a39c846c9acc0c1c9270f6b5e (127.25.109.1:33745)
WARNING: no policy specified for kudu/127.25.109.2@KRBTEST.COM; defaulting to no policy
I20260924 16:04:38.111948 7504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.1@KRBTEST.COM'} at 127.25.109.1:41425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
I20260924 16:04:38.129761 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:0
--local_ip_for_outbound_sockets=127.25.109.2
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:42581
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:41887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:38.241104 7705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:38.241346 7705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:38.241387 7705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:38.245404 7705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:38.245455 7705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:38.245515 7705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:04:38.250630 7705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:41887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:42581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:04:38.251557 7705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:38.252583 7705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:38.261768 7713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:38.261790 7711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:38.261788 7710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:38.262151 7705 server_base.cc:1034] running on GCE node
I20260924 16:04:38.262519 7705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:38.263067 7705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:38.264261 7705 hybrid_clock.cc:648] HybridClock initialized: now 1790265878264251 us; error 31 us; skew 500 ppm
Sep 24 16:04:38 dist-test-slave-n106 krb5kdc[7459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265878, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:38.266957 7705 init.cc:377] Logged in from keytab as kudu/127.25.109.2@KRBTEST.COM (short username kudu)
I20260924 16:04:38.267756 7705 webserver.cc:460] Webserver started at http://127.25.109.2:39489/ using document root <none> and password file <none>
I20260924 16:04:38.268309 7705 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:38.268399 7705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:38.268656 7705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:38.270627 7705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/data/instance:
uuid: "7ee64b2ecdbd495295a0dbc199c41f57"
format_stamp: "Formatted at 2026-09-24 16:04:38 on dist-test-slave-n106"
server_key: "76a22351cc78d1e13d487bdeb99bd648"
server_key_iv: "1b1be215adc5f8db2635f225265ee31f"
server_key_version: "encryptionkey@0"
I20260924 16:04:38.271186 7705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance:
uuid: "7ee64b2ecdbd495295a0dbc199c41f57"
format_stamp: "Formatted at 2026-09-24 16:04:38 on dist-test-slave-n106"
server_key: "76a22351cc78d1e13d487bdeb99bd648"
server_key_iv: "1b1be215adc5f8db2635f225265ee31f"
server_key_version: "encryptionkey@0"
I20260924 16:04:38.274821 7705 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 16:04:38.277314 7720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:38.278540 7705 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:04:38.278673 7705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "7ee64b2ecdbd495295a0dbc199c41f57"
format_stamp: "Formatted at 2026-09-24 16:04:38 on dist-test-slave-n106"
server_key: "76a22351cc78d1e13d487bdeb99bd648"
server_key_iv: "1b1be215adc5f8db2635f225265ee31f"
server_key_version: "encryptionkey@0"
I20260924 16:04:38.278782 7705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:38.298372 7705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:38.301610 7705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:38.301831 7705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:38.302443 7705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:38.303313 7705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:38.303385 7705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:38.303470 7705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:38.303521 7705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:38.312487 7705 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:42331
I20260924 16:04:38.312551 7835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:42331 every 8 connection(s)
I20260924 16:04:38.313437 7705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:04:38.316197 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 7705
I20260924 16:04:38.316346 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance
I20260924 16:04:38.316699 26036 external_mini_cluster.cc:1428] Setting key 5c88097be652fbcb176251f493b1fc62
Sep 24 16:04:38 dist-test-slave-n106 krb5kdc[7459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265878, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.109.3@KRBTEST.COM; defaulting to no policy
I20260924 16:04:38.329154 7543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:38.315376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:37989 (local address 127.25.109.62:42581)
0924 16:04:38.315544 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:38.315549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:38.316719 (+ 1170us) server_negotiation.cc:407] Connection header received
0924 16:04:38.318460 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:38.318462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:38.318527 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:38.318586 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:38.320212 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.321123 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.322249 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.322670 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.322791 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:38.325553 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:38.325559 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:38.325563 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:38.325597 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:38.327216 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:38.327705 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:38.327708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:38.327709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:38.327785 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:38.328124 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:38.328127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:38.328129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:38.328323 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:38.328400 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:38.328922 (+ 522us) server_negotiation.cc:299] Negotiation successful
0924 16:04:38.329028 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.3@KRBTEST.COM" created.
I20260924 16:04:38.330281 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:38.316012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.2:37989)
0924 16:04:38.316555 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:38.316614 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:38.318204 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:38.318779 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:38.318792 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:38.319332 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:38.320027 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:38.320052 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.321254 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.321257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:38.322116 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:38.322135 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.322830 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.322833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:38.322976 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:38.323659 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:38.323675 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:38.325363 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:38.327334 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:38.327339 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:38.327351 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:38.327603 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:38.327866 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:38.327868 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:38.327870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:38.327997 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:38.328455 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:38.328467 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:38.328708 (+ 241us) client_negotiation.cc:769] Sending connection context
0924 16:04:38.328868 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 16:04:38.329211 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":124,"threads_started":1}
I20260924 16:04:38.331543 7836 heartbeater.cc:347] Connected to a master server at 127.25.109.62:42581
I20260924 16:04:38.331802 7836 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:38.332309 7836 heartbeater.cc:511] Master 127.25.109.62:42581 requested a full tablet report, sending...
I20260924 16:04:38.333431 7504 ts_manager.cc:194] Registered new tserver with Master: 7ee64b2ecdbd495295a0dbc199c41f57 (127.25.109.2:42331)
I20260924 16:04:38.334010 7504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.2@KRBTEST.COM'} at 127.25.109.2:37989
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
I20260924 16:04:38.347002 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:0
--local_ip_for_outbound_sockets=127.25.109.3
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:42581
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:41887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:38.461516 7841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:38.461846 7841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:38.461949 7841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:38.466140 7841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:38.466231 7841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:38.466325 7841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:04:38.471997 7841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:41887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:42581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:04:38.473233 7841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:38.474315 7841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:38.484581 7847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:38.484619 7846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:38.484699 7841 server_base.cc:1034] running on GCE node
W20260924 16:04:38.484616 7849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:38.485256 7841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:38.485937 7841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:38.487112 7841 hybrid_clock.cc:648] HybridClock initialized: now 1790265878487095 us; error 28 us; skew 500 ppm
Sep 24 16:04:38 dist-test-slave-n106 krb5kdc[7459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265878, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:38.490039 7841 init.cc:377] Logged in from keytab as kudu/127.25.109.3@KRBTEST.COM (short username kudu)
I20260924 16:04:38.490970 7841 webserver.cc:460] Webserver started at http://127.25.109.3:45117/ using document root <none> and password file <none>
I20260924 16:04:38.491600 7841 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:38.491665 7841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:38.491909 7841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:38.494191 7841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/data/instance:
uuid: "d63de18a948845c6b7e1dc33a4192d45"
format_stamp: "Formatted at 2026-09-24 16:04:38 on dist-test-slave-n106"
server_key: "f65498179a7936acc8fcffdf5b21eab0"
server_key_iv: "8f4c22597cf1361ca8ec46f7625cead1"
server_key_version: "encryptionkey@0"
I20260924 16:04:38.494801 7841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance:
uuid: "d63de18a948845c6b7e1dc33a4192d45"
format_stamp: "Formatted at 2026-09-24 16:04:38 on dist-test-slave-n106"
server_key: "f65498179a7936acc8fcffdf5b21eab0"
server_key_iv: "8f4c22597cf1361ca8ec46f7625cead1"
server_key_version: "encryptionkey@0"
I20260924 16:04:38.498804 7841 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:04:38.501397 7856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:38.502652 7841 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:04:38.502810 7841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "d63de18a948845c6b7e1dc33a4192d45"
format_stamp: "Formatted at 2026-09-24 16:04:38 on dist-test-slave-n106"
server_key: "f65498179a7936acc8fcffdf5b21eab0"
server_key_iv: "8f4c22597cf1361ca8ec46f7625cead1"
server_key_version: "encryptionkey@0"
I20260924 16:04:38.502947 7841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:38.515805 7841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:38.518986 7841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:38.519150 7841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:38.519716 7841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:38.520584 7841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:38.520637 7841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:38.520679 7841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:38.520695 7841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:38.529968 7841 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:40103
I20260924 16:04:38.529994 7973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:40103 every 8 connection(s)
I20260924 16:04:38.530897 7841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:04:38.533670 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 7841
I20260924 16:04:38.533803 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance
I20260924 16:04:38.534183 26036 external_mini_cluster.cc:1428] Setting key dc7eb23db0531c86e2d6d5f5710bc09a
Sep 24 16:04:38 dist-test-slave-n106 krb5kdc[7459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265878, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:04:38.549650 7543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:38.532783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:56691 (local address 127.25.109.62:42581)
0924 16:04:38.532931 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:38.532935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:38.533873 (+ 938us) server_negotiation.cc:407] Connection header received
0924 16:04:38.535891 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:38.535895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:38.536014 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:38.536100 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:38.537903 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.539047 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.541148 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.541387 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.541502 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:38.544831 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:38.544837 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:38.544843 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:38.544875 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:38.546799 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:38.547566 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:38.547571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:38.547574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:38.547693 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:38.548149 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:38.548153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:38.548155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:38.548398 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:38.548657 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:38.549344 (+ 687us) server_negotiation.cc:299] Negotiation successful
0924 16:04:38.549466 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 16:04:38.550801 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:38.533173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.3:56691)
0924 16:04:38.533703 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:38.533764 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:38.535607 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:38.536340 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:38.536352 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:38.536961 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:38.537735 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:38.537761 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.539189 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.539196 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:38.541006 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:38.541018 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.541540 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.541544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:38.541905 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:38.542494 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:38.542510 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:38.544660 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:38.546955 (+ 2295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:38.546961 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:38.546999 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:38.547429 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:38.547801 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:38.547804 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:38.547807 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:38.548024 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:38.548729 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:38.548738 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:38.549066 (+ 328us) client_negotiation.cc:769] Sending connection context
0924 16:04:38.549272 (+ 206us) client_negotiation.cc:240] Negotiation successful
0924 16:04:38.549608 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":151,"threads_started":1}
I20260924 16:04:38.552201 7974 heartbeater.cc:347] Connected to a master server at 127.25.109.62:42581
I20260924 16:04:38.552529 7974 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:38.553171 7974 heartbeater.cc:511] Master 127.25.109.62:42581 requested a full tablet report, sending...
I20260924 16:04:38.554464 7504 ts_manager.cc:194] Registered new tserver with Master: d63de18a948845c6b7e1dc33a4192d45 (127.25.109.3:40103)
I20260924 16:04:38.555171 7504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.3@KRBTEST.COM'} at 127.25.109.3:56691
I20260924 16:04:38.561794 26036 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:04:38.573166 7543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:38.564298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42874 (local address 127.25.109.62:42581)
0924 16:04:38.564465 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:38.564470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:38.564655 (+ 185us) server_negotiation.cc:407] Connection header received
0924 16:04:38.564774 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:38.564777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:38.564825 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:38.564911 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:38.565888 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.566408 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.567851 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.568075 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.568188 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:38.569669 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:38.569674 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:38.569676 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:38.569710 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:38.571384 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:38.571823 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:38.571827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:38.571829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:38.571915 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:38.572196 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:38.572201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:38.572202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:38.572506 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:38.572658 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:38.572884 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 16:04:38.572999 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:04:38.576690 7504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:04:38.582173 7504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:04:38.597433 7986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:38.591163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:40103 (local address 127.0.0.1:47770)
0924 16:04:38.591564 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:38.591623 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:38.591756 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:38.592766 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:38.592773 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:38.592801 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:38.592961 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:38.592968 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.594625 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.594629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:38.595969 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:38.595976 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.597066 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.597069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:38.597155 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:38.597176 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 16:04:38.597234 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 16:04:38.597287 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":79,"thread_start_us":93,"threads_started":1}
I20260924 16:04:38.597537 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:38.591337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47770 (local address 127.25.109.3:40103)
0924 16:04:38.592198 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:38.592212 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:38.592237 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:04:38.592317 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:38.592323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:38.592575 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:38.593117 (+ 542us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:38.593300 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.594480 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.596103 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.596957 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.597063 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:38.597199 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:38.597261 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 16:04:38.597356 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743,"thread_start_us":138,"threads_started":1}
I20260924 16:04:38.597340 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:38.591521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:33745 (local address 127.0.0.1:36816)
0924 16:04:38.592450 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:38.592465 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:38.592558 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:38.593088 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:38.593091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:38.593108 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:38.593220 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:38.593227 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.594356 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.594362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:38.595676 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:38.595686 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.596785 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.596788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:38.596907 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:38.596945 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 16:04:38.597055 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 16:04:38.597170 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"mutex_wait_us":45,"thread_start_us":95,"threads_started":1}
I20260924 16:04:38.598095 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:38.591997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36816 (local address 127.25.109.1:33745)
0924 16:04:38.592619 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:38.592636 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:38.592662 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:04:38.592741 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:38.592746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:38.592945 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:38.593120 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:38.593349 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.594225 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.595804 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.596661 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.596784 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:38.596942 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:38.597429 (+ 487us) server_negotiation.cc:299] Negotiation successful
0924 16:04:38.597539 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":92,"threads_started":1}
I20260924 16:04:38.598388 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:38.592311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42694 (local address 127.25.109.2:42331)
0924 16:04:38.592763 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:38.592775 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:38.592844 (+ 69us) server_negotiation.cc:407] Connection header received
0924 16:04:38.592999 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:38.593007 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:38.593194 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:38.593341 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:38.593543 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.594642 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.596103 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.596957 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.597064 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:38.597199 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:38.598000 (+ 801us) server_negotiation.cc:299] Negotiation successful
0924 16:04:38.598086 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":143,"threads_started":1}
I20260924 16:04:38.598721 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:38.591901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:42331 (local address 127.0.0.1:42694)
0924 16:04:38.592764 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:38.592776 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:38.592870 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:38.593315 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:38.593318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:38.593335 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:38.593425 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:38.593430 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.594773 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.594776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:38.595961 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:38.595972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.597265 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.597270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:38.597365 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:38.597387 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 16:04:38.597457 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 16:04:38.598584 (+ 1127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":28,"spinlock_wait_cycles":1280,"thread_start_us":103,"threads_started":1}
I20260924 16:04:38.600620 7632 tablet_service.cc:1467] Processing CreateTablet for tablet 7a3ccef5e82641dca568446be08eeca0 (DEFAULT_TABLE table=test-table [id=841ba3e84ad2405bb2762729129d33dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:04:38.600622 7904 tablet_service.cc:1467] Processing CreateTablet for tablet 7a3ccef5e82641dca568446be08eeca0 (DEFAULT_TABLE table=test-table [id=841ba3e84ad2405bb2762729129d33dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:04:38.601207 7770 tablet_service.cc:1467] Processing CreateTablet for tablet 7a3ccef5e82641dca568446be08eeca0 (DEFAULT_TABLE table=test-table [id=841ba3e84ad2405bb2762729129d33dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:04:38.601598 7632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a3ccef5e82641dca568446be08eeca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:38.601634 7904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a3ccef5e82641dca568446be08eeca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:38.602075 7770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a3ccef5e82641dca568446be08eeca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:38.607970 7992 tablet_bootstrap.cc:492] T 7a3ccef5e82641dca568446be08eeca0 P ccf0f95a39c846c9acc0c1c9270f6b5e: Bootstrap starting.
I20260924 16:04:38.609280 7993 tablet_bootstrap.cc:492] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57: Bootstrap starting.
I20260924 16:04:38.610118 7992 tablet_bootstrap.cc:654] T 7a3ccef5e82641dca568446be08eeca0 P ccf0f95a39c846c9acc0c1c9270f6b5e: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:38.610828 7992 log.cc:824] T 7a3ccef5e82641dca568446be08eeca0 P ccf0f95a39c846c9acc0c1c9270f6b5e: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:38.611408 7994 tablet_bootstrap.cc:492] T 7a3ccef5e82641dca568446be08eeca0 P d63de18a948845c6b7e1dc33a4192d45: Bootstrap starting.
I20260924 16:04:38.611433 7993 tablet_bootstrap.cc:654] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:38.612249 7993 log.cc:824] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:38.612932 7992 tablet_bootstrap.cc:492] T 7a3ccef5e82641dca568446be08eeca0 P ccf0f95a39c846c9acc0c1c9270f6b5e: No bootstrap required, opened a new log
I20260924 16:04:38.613118 7992 ts_tablet_manager.cc:1397] T 7a3ccef5e82641dca568446be08eeca0 P ccf0f95a39c846c9acc0c1c9270f6b5e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 16:04:38.613320 7994 tablet_bootstrap.cc:654] T 7a3ccef5e82641dca568446be08eeca0 P d63de18a948845c6b7e1dc33a4192d45: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:38.614161 7993 tablet_bootstrap.cc:492] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57: No bootstrap required, opened a new log
I20260924 16:04:38.614357 7993 ts_tablet_manager.cc:1397] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 16:04:38.614493 7994 log.cc:824] T 7a3ccef5e82641dca568446be08eeca0 P d63de18a948845c6b7e1dc33a4192d45: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:38.616273 7994 tablet_bootstrap.cc:492] T 7a3ccef5e82641dca568446be08eeca0 P d63de18a948845c6b7e1dc33a4192d45: No bootstrap required, opened a new log
I20260924 16:04:38.616482 7994 ts_tablet_manager.cc:1397] T 7a3ccef5e82641dca568446be08eeca0 P d63de18a948845c6b7e1dc33a4192d45: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 16:04:38.617599 7992 raft_consensus.cc:348] T 7a3ccef5e82641dca568446be08eeca0 P ccf0f95a39c846c9acc0c1c9270f6b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63de18a948845c6b7e1dc33a4192d45" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40103 } } peers { permanent_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 42331 } } peers { permanent_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33745 } }
I20260924 16:04:38.617865 7992 raft_consensus.cc:374] T 7a3ccef5e82641dca568446be08eeca0 P ccf0f95a39c846c9acc0c1c9270f6b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:38.617925 7992 raft_consensus.cc:729] T 7a3ccef5e82641dca568446be08eeca0 P ccf0f95a39c846c9acc0c1c9270f6b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf0f95a39c846c9acc0c1c9270f6b5e, State: Initialized, Role: FOLLOWER
I20260924 16:04:38.618146 7993 raft_consensus.cc:348] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63de18a948845c6b7e1dc33a4192d45" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40103 } } peers { permanent_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 42331 } } peers { permanent_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33745 } }
I20260924 16:04:38.618374 7993 raft_consensus.cc:374] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:38.618432 7993 raft_consensus.cc:729] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ee64b2ecdbd495295a0dbc199c41f57, State: Initialized, Role: FOLLOWER
I20260924 16:04:38.618362 7992 consensus_queue.cc:260] T 7a3ccef5e82641dca568446be08eeca0 P ccf0f95a39c846c9acc0c1c9270f6b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63de18a948845c6b7e1dc33a4192d45" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40103 } } peers { permanent_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 42331 } } peers { permanent_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33745 } }
I20260924 16:04:38.618860 7993 consensus_queue.cc:260] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63de18a948845c6b7e1dc33a4192d45" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40103 } } peers { permanent_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 42331 } } peers { permanent_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33745 } }
I20260924 16:04:38.619354 7992 ts_tablet_manager.cc:1428] T 7a3ccef5e82641dca568446be08eeca0 P ccf0f95a39c846c9acc0c1c9270f6b5e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 16:04:38.619692 7700 heartbeater.cc:503] Master 127.25.109.62:42581 was elected leader, sending a full tablet report...
I20260924 16:04:38.619757 7993 ts_tablet_manager.cc:1428] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 16:04:38.620082 7836 heartbeater.cc:503] Master 127.25.109.62:42581 was elected leader, sending a full tablet report...
I20260924 16:04:38.620604 7994 raft_consensus.cc:348] T 7a3ccef5e82641dca568446be08eeca0 P d63de18a948845c6b7e1dc33a4192d45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63de18a948845c6b7e1dc33a4192d45" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40103 } } peers { permanent_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 42331 } } peers { permanent_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33745 } }
I20260924 16:04:38.620885 7994 raft_consensus.cc:374] T 7a3ccef5e82641dca568446be08eeca0 P d63de18a948845c6b7e1dc33a4192d45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:38.620949 7994 raft_consensus.cc:729] T 7a3ccef5e82641dca568446be08eeca0 P d63de18a948845c6b7e1dc33a4192d45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d63de18a948845c6b7e1dc33a4192d45, State: Initialized, Role: FOLLOWER
I20260924 16:04:38.621332 7994 consensus_queue.cc:260] T 7a3ccef5e82641dca568446be08eeca0 P d63de18a948845c6b7e1dc33a4192d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63de18a948845c6b7e1dc33a4192d45" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40103 } } peers { permanent_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 42331 } } peers { permanent_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33745 } }
I20260924 16:04:38.622148 7994 ts_tablet_manager.cc:1428] T 7a3ccef5e82641dca568446be08eeca0 P d63de18a948845c6b7e1dc33a4192d45: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260924 16:04:38.622929 7974 heartbeater.cc:503] Master 127.25.109.62:42581 was elected leader, sending a full tablet report...
I20260924 16:04:38.658233 7999 raft_consensus.cc:482] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:04:38.658519 7999 raft_consensus.cc:504] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63de18a948845c6b7e1dc33a4192d45" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40103 } } peers { permanent_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 42331 } } peers { permanent_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33745 } }
I20260924 16:04:38.659746 7999 leader_election.cc:288] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d63de18a948845c6b7e1dc33a4192d45 (127.25.109.3:40103), ccf0f95a39c846c9acc0c1c9270f6b5e (127.25.109.1:33745)
I20260924 16:04:38.663743 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:38.660060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:38723 (local address 127.25.109.3:40103)
0924 16:04:38.660192 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:38.660196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:38.660212 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:04:38.660273 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:38.660277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:38.660329 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:38.660436 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:38.660852 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.661469 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.662688 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.663144 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.663289 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:38.663348 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:38.663551 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 16:04:38.663623 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:04:38.663743 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:38.659932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:40103 (local address 127.25.109.2:38723)
0924 16:04:38.660047 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:38.660064 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:38.660162 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:38.660507 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:38.660517 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:38.660555 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:38.660716 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:38.660723 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.661598 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.661601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:38.662515 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:38.662525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.663278 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.663282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:38.663396 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:38.663437 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 16:04:38.663534 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 16:04:38.663605 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":58,"spinlock_wait_cycles":4992}
I20260924 16:04:38.663883 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:38.660263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:43469 (local address 127.25.109.1:33745)
0924 16:04:38.660466 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:38.660472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:38.660557 (+ 85us) server_negotiation.cc:407] Connection header received
0924 16:04:38.660686 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:38.660690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:38.660737 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:38.660812 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:38.661079 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.661671 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.662853 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.663277 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.663400 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:38.663466 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:38.663655 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 16:04:38.663741 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 16:04:38.663972 8001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:38.659987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:33745 (local address 127.25.109.2:43469)
0924 16:04:38.660460 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:38.660474 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:38.660585 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:38.660837 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:38.660840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:38.660856 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:38.660973 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:38.660979 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.661771 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.661774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:38.662728 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:38.662738 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.663392 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.663396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:38.663515 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:38.663537 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 16:04:38.663630 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 16:04:38.663701 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":1,"thread_start_us":121,"threads_started":1}
I20260924 16:04:38.664387 7926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3ccef5e82641dca568446be08eeca0" candidate_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d63de18a948845c6b7e1dc33a4192d45" is_pre_election: true
I20260924 16:04:38.664779 7926 raft_consensus.cc:2454] T 7a3ccef5e82641dca568446be08eeca0 P d63de18a948845c6b7e1dc33a4192d45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ee64b2ecdbd495295a0dbc199c41f57 in term 0.
I20260924 16:04:38.664744 7653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3ccef5e82641dca568446be08eeca0" candidate_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" is_pre_election: true
I20260924 16:04:38.665021 7653 raft_consensus.cc:2454] T 7a3ccef5e82641dca568446be08eeca0 P ccf0f95a39c846c9acc0c1c9270f6b5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ee64b2ecdbd495295a0dbc199c41f57 in term 0.
I20260924 16:04:38.665313 7721 leader_election.cc:302] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ee64b2ecdbd495295a0dbc199c41f57, d63de18a948845c6b7e1dc33a4192d45; no voters:
I20260924 16:04:38.665594 7999 raft_consensus.cc:2781] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:04:38.665688 7999 raft_consensus.cc:482] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:04:38.665740 7999 raft_consensus.cc:3037] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:38.666939 7999 raft_consensus.cc:504] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63de18a948845c6b7e1dc33a4192d45" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40103 } } peers { permanent_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 42331 } } peers { permanent_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33745 } }
I20260924 16:04:38.667316 7999 leader_election.cc:288] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57 [CANDIDATE]: Term 1 election: Requested vote from peers d63de18a948845c6b7e1dc33a4192d45 (127.25.109.3:40103), ccf0f95a39c846c9acc0c1c9270f6b5e (127.25.109.1:33745)
I20260924 16:04:38.667599 7926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3ccef5e82641dca568446be08eeca0" candidate_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d63de18a948845c6b7e1dc33a4192d45"
I20260924 16:04:38.667727 7926 raft_consensus.cc:3037] T 7a3ccef5e82641dca568446be08eeca0 P d63de18a948845c6b7e1dc33a4192d45 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:38.667734 7653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3ccef5e82641dca568446be08eeca0" candidate_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e"
I20260924 16:04:38.667867 7653 raft_consensus.cc:3037] T 7a3ccef5e82641dca568446be08eeca0 P ccf0f95a39c846c9acc0c1c9270f6b5e [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:38.668924 7926 raft_consensus.cc:2454] T 7a3ccef5e82641dca568446be08eeca0 P d63de18a948845c6b7e1dc33a4192d45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ee64b2ecdbd495295a0dbc199c41f57 in term 1.
I20260924 16:04:38.668970 7653 raft_consensus.cc:2454] T 7a3ccef5e82641dca568446be08eeca0 P ccf0f95a39c846c9acc0c1c9270f6b5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ee64b2ecdbd495295a0dbc199c41f57 in term 1.
I20260924 16:04:38.669219 7721 leader_election.cc:302] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ee64b2ecdbd495295a0dbc199c41f57, d63de18a948845c6b7e1dc33a4192d45; no voters:
I20260924 16:04:38.669404 7999 raft_consensus.cc:2781] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:38.669610 7999 raft_consensus.cc:686] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57 [term 1 LEADER]: Becoming Leader. State: Replica: 7ee64b2ecdbd495295a0dbc199c41f57, State: Running, Role: LEADER
I20260924 16:04:38.669938 7999 consensus_queue.cc:237] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63de18a948845c6b7e1dc33a4192d45" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40103 } } peers { permanent_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 42331 } } peers { permanent_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33745 } }
I20260924 16:04:38.673134 7502 catalog_manager.cc:5495] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ee64b2ecdbd495295a0dbc199c41f57 (127.25.109.2). New cstate: current_term: 1 leader_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63de18a948845c6b7e1dc33a4192d45" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 42331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33745 } health_report { overall_health: UNKNOWN } } }
I20260924 16:04:38.693547 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:38.690260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42706 (local address 127.25.109.2:42331)
0924 16:04:38.690389 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:38.690393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:38.690406 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:04:38.690519 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:38.690521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:38.690576 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:38.690662 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:04:38.690883 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.691399 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.692215 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:38.692490 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:38.692574 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:38.692836 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:04:38.693242 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:04:38.693329 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:38.693387 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 16:04:38.693436 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 16:04:38.699345 7926 raft_consensus.cc:1261] T 7a3ccef5e82641dca568446be08eeca0 P d63de18a948845c6b7e1dc33a4192d45 [term 1 FOLLOWER]: Refusing update from remote peer 7ee64b2ecdbd495295a0dbc199c41f57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:04:38.699354 7653 raft_consensus.cc:1261] T 7a3ccef5e82641dca568446be08eeca0 P ccf0f95a39c846c9acc0c1c9270f6b5e [term 1 FOLLOWER]: Refusing update from remote peer 7ee64b2ecdbd495295a0dbc199c41f57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:04:38.700066 7999 consensus_queue.cc:1035] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d63de18a948845c6b7e1dc33a4192d45" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:04:38.700263 8002 consensus_queue.cc:1035] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:04:38.706569 8006 mvcc.cc:205] Tried to move back new op lower bound from 7332929039145689088 to 7332929039034490880. Current Snapshot: MvccSnapshot[applied={T|T < 7332929039145689088}]
I20260924 16:04:38.707300 8007 mvcc.cc:205] Tried to move back new op lower bound from 7332929039145689088 to 7332929039034490880. Current Snapshot: MvccSnapshot[applied={T|T < 7332929039145689088}]
I20260924 16:04:38.716187 7502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:04:38.716342 7502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:04:38.718894 7502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67: Sending DeleteTablet for 3 replicas of tablet 7a3ccef5e82641dca568446be08eeca0
I20260924 16:04:38.719816 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 7566
I20260924 16:04:38.719779 7904 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a3ccef5e82641dca568446be08eeca0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:38 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:47770
I20260924 16:04:38.719784 7632 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a3ccef5e82641dca568446be08eeca0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:38 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:36816
I20260924 16:04:38.721117 7770 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a3ccef5e82641dca568446be08eeca0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:38 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:42694
I20260924 16:04:38.721205 8016 tablet_replica.cc:330] T 7a3ccef5e82641dca568446be08eeca0 P d63de18a948845c6b7e1dc33a4192d45: stopping tablet replica
I20260924 16:04:38.721551 8016 raft_consensus.cc:2229] T 7a3ccef5e82641dca568446be08eeca0 P d63de18a948845c6b7e1dc33a4192d45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:04:38.721877 8016 raft_consensus.cc:2258] T 7a3ccef5e82641dca568446be08eeca0 P d63de18a948845c6b7e1dc33a4192d45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:04:38.723215 8016 ts_tablet_manager.cc:1905] T 7a3ccef5e82641dca568446be08eeca0 P d63de18a948845c6b7e1dc33a4192d45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:04:38.722347 8017 tablet_replica.cc:330] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57: stopping tablet replica
I20260924 16:04:38.724860 8017 raft_consensus.cc:2229] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:04:38.725356 8017 raft_consensus.cc:2258] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:04:38.726233 8016 ts_tablet_manager.cc:1918] T 7a3ccef5e82641dca568446be08eeca0 P d63de18a948845c6b7e1dc33a4192d45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:04:38.726336 8016 log.cc:1196] T 7a3ccef5e82641dca568446be08eeca0 P d63de18a948845c6b7e1dc33a4192d45: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/wal/wals/7a3ccef5e82641dca568446be08eeca0
I20260924 16:04:38.726701 8016 ts_tablet_manager.cc:1939] T 7a3ccef5e82641dca568446be08eeca0 P d63de18a948845c6b7e1dc33a4192d45: Deleting consensus metadata
I20260924 16:04:38.727090 8017 ts_tablet_manager.cc:1905] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:04:38.727993 7488 catalog_manager.cc:4841] TS d63de18a948845c6b7e1dc33a4192d45 (127.25.109.3:40103): tablet 7a3ccef5e82641dca568446be08eeca0 (table test-table [id=841ba3e84ad2405bb2762729129d33dd]) successfully deleted
W20260924 16:04:38.729403 7490 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.109.1:33745 (error 108)
I20260924 16:04:38.729671 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 7705
I20260924 16:04:38.731351 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:38.730820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:33745 (local address 127.0.0.1:36832)
0924 16:04:38.731190 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:38.731264 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.1:33745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254}
W20260924 16:04:38.731516 7490 catalog_manager.cc:4584] TS ccf0f95a39c846c9acc0c1c9270f6b5e (127.25.109.1:33745): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7a3ccef5e82641dca568446be08eeca0: Network error: Client connection negotiation failed: client connection to 127.25.109.1:33745: connect: Connection refused (error 111)
I20260924 16:04:38.736007 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 7841
I20260924 16:04:38.736227 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:38.735950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:42331 (local address 127.0.0.1:42708)
0924 16:04:38.736073 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:38.736134 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.2:42331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 16:04:38.742559 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 7472
I20260924 16:04:38.749264 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:42581
--webserver_interface=127.25.109.62
--webserver_port=38041
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:41887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:42581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:38.859051 8019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:38.859310 8019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:38.859386 8019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:38.863512 8019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:04:38.863562 8019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:38.863581 8019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:38.863632 8019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:04:38.863662 8019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:04:38.868912 8019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:41887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:42581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:42581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=38041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:04:38.869915 8019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:38.870940 8019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:38.884712 8027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:38.884771 8029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:38.884881 8019 server_base.cc:1034] running on GCE node
W20260924 16:04:38.884723 8026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:38.885658 8019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:38.886802 8019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:38.887964 8019 hybrid_clock.cc:648] HybridClock initialized: now 1790265878887947 us; error 28 us; skew 500 ppm
Sep 24 16:04:38 dist-test-slave-n106 krb5kdc[7459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265878, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:38.890712 8019 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:04:38.891650 8019 webserver.cc:460] Webserver started at http://127.25.109.62:38041/ using document root <none> and password file <none>
I20260924 16:04:38.892275 8019 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:38.892391 8019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:38.896533 8019 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260924 16:04:38.898816 8036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:38.900017 8019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:04:38.900182 8019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "38fa2227488d452a92f88ae33c25bf67"
format_stamp: "Formatted at 2026-09-24 16:04:37 on dist-test-slave-n106"
server_key: "17b0a5cdb7457c565552f518246b0f56"
server_key_iv: "43045f6b8bb5329960d699a2ffb73e96"
server_key_version: "encryptionkey@0"
I20260924 16:04:38.900861 8019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:38.926110 8019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:38.929241 8019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:38.929466 8019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:38.936853 8019 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:42581
I20260924 16:04:38.937155 8090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:42581 every 8 connection(s)
I20260924 16:04:38.938066 8019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:04:38.942498 8091 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:04:38.944168 8091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67: Bootstrap starting.
I20260924 16:04:38.945688 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 8019
I20260924 16:04:38.946152 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:33745
--local_ip_for_outbound_sockets=127.25.109.1
--tserver_master_addrs=127.25.109.62:42581
--webserver_port=44365
--webserver_interface=127.25.109.1
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:41887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:04:38.961443 8091 log.cc:824] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:38.970252 8091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:04:38.970620 8091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67: Bootstrap complete.
I20260924 16:04:38.974673 8091 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fa2227488d452a92f88ae33c25bf67" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 42581 } }
I20260924 16:04:38.975095 8091 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38fa2227488d452a92f88ae33c25bf67, State: Initialized, Role: FOLLOWER
I20260924 16:04:38.975579 8091 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fa2227488d452a92f88ae33c25bf67" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 42581 } }
I20260924 16:04:38.975740 8091 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:04:38.975824 8091 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:04:38.975946 8091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [term 1 FOLLOWER]: Advancing to term 2
I20260924 16:04:38.977921 8091 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fa2227488d452a92f88ae33c25bf67" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 42581 } }
I20260924 16:04:38.978269 8091 leader_election.cc:302] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38fa2227488d452a92f88ae33c25bf67; no voters:
I20260924 16:04:38.978572 8091 leader_election.cc:288] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 16:04:38.978706 8095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 16:04:38.978972 8095 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [term 2 LEADER]: Becoming Leader. State: Replica: 38fa2227488d452a92f88ae33c25bf67, State: Running, Role: LEADER
I20260924 16:04:38.979290 8095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fa2227488d452a92f88ae33c25bf67" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 42581 } }
I20260924 16:04:38.979774 8091 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:04:38.980806 8096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "38fa2227488d452a92f88ae33c25bf67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fa2227488d452a92f88ae33c25bf67" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 42581 } } }
I20260924 16:04:38.980942 8096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:38.981601 8097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38fa2227488d452a92f88ae33c25bf67. Latest consensus state: current_term: 2 leader_uuid: "38fa2227488d452a92f88ae33c25bf67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fa2227488d452a92f88ae33c25bf67" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 42581 } } }
I20260924 16:04:38.981693 8097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:38.984508 8112 catalog_manager.cc:1259] Loaded cluster ID: d5a4fc66d2284c1299703c4863799ad0
I20260924 16:04:38.984570 8112 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67: loading cluster ID for follower catalog manager: success
W20260924 16:04:38.985347 8112 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260924 16:04:38.985432 8112 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260924 16:04:38.985498 8105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:04:38.987339 8105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:04:38.987772 8105 catalog_manager.cc:1259] Loaded cluster ID: d5a4fc66d2284c1299703c4863799ad0
I20260924 16:04:38.987846 8105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:04:38.988722 8105 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 16:04:38.991715 8105 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 16:04:38.995939 8105 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:04:38.996784 8105 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67: Loaded TSK: 0
I20260924 16:04:38.997737 8105 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 16:04:39.075171 8093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:39.075466 8093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:39.075572 8093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:39.079708 8093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:39.079792 8093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:39.079882 8093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:04:39.085603 8093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:41887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:33745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=44365
--tserver_master_addrs=127.25.109.62:42581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:04:39.086628 8093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:39.087639 8093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:39.097126 8119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:39.097128 8121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:39.097220 8123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:39.098104 8093 server_base.cc:1034] running on GCE node
I20260924 16:04:39.098507 8093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:39.099104 8093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:39.100296 8093 hybrid_clock.cc:648] HybridClock initialized: now 1790265879100270 us; error 42 us; skew 500 ppm
Sep 24 16:04:39 dist-test-slave-n106 krb5kdc[7459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265879, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:39.103057 8093 init.cc:377] Logged in from keytab as kudu/127.25.109.1@KRBTEST.COM (short username kudu)
I20260924 16:04:39.104010 8093 webserver.cc:460] Webserver started at http://127.25.109.1:44365/ using document root <none> and password file <none>
I20260924 16:04:39.104646 8093 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:39.104869 8093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:39.108951 8093 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:04:39.111117 8130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:39.112529 8093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:04:39.112643 8093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e"
format_stamp: "Formatted at 2026-09-24 16:04:38 on dist-test-slave-n106"
server_key: "4017e7eca347eb3faa717f9498fc91af"
server_key_iv: "638ec19e00271cde59e6e735835257f0"
server_key_version: "encryptionkey@0"
I20260924 16:04:39.113170 8093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:39.136283 8093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:39.139629 8093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:39.139861 8093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:39.140524 8093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:39.141641 8137 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 16:04:39.146004 8093 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 16:04:39.146065 8093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260924 16:04:39.146145 8093 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 16:04:39.147881 8093 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 16:04:39.147927 8093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 16:04:39.148254 8137 tablet_bootstrap.cc:492] T 7a3ccef5e82641dca568446be08eeca0 P ccf0f95a39c846c9acc0c1c9270f6b5e: Bootstrap starting.
I20260924 16:04:39.158550 8093 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:33745
I20260924 16:04:39.158643 8250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:33745 every 8 connection(s)
I20260924 16:04:39.159579 8093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:04:39.163064 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 8093
I20260924 16:04:39.163583 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:42331
--local_ip_for_outbound_sockets=127.25.109.2
--tserver_master_addrs=127.25.109.62:42581
--webserver_port=39489
--webserver_interface=127.25.109.2
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:41887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:04:39.168008 8137 log.cc:824] T 7a3ccef5e82641dca568446be08eeca0 P ccf0f95a39c846c9acc0c1c9270f6b5e: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:04:39 dist-test-slave-n106 krb5kdc[7459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265879, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:04:39.176466 8137 tablet_bootstrap.cc:492] T 7a3ccef5e82641dca568446be08eeca0 P ccf0f95a39c846c9acc0c1c9270f6b5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:04:39.176944 8137 tablet_bootstrap.cc:492] T 7a3ccef5e82641dca568446be08eeca0 P ccf0f95a39c846c9acc0c1c9270f6b5e: Bootstrap complete.
I20260924 16:04:39.178288 8137 ts_tablet_manager.cc:1397] T 7a3ccef5e82641dca568446be08eeca0 P ccf0f95a39c846c9acc0c1c9270f6b5e: Time spent bootstrapping tablet: real 0.030s user 0.024s sys 0.004s
I20260924 16:04:39.179836 8254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:39.161646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.1:38065)
0924 16:04:39.162343 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:39.162397 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:39.163479 (+ 1082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:39.164777 (+ 1298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:39.164784 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:39.165304 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:39.165974 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:39.165997 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.167803 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.167809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:39.169444 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:39.169459 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.170556 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.170561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:39.170809 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:39.171572 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:39.171600 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:39.173662 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:39.176314 (+ 2652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:39.176320 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:39.176334 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:39.176718 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:39.177163 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:39.177166 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:39.177168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:39.177338 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:39.178073 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:39.178079 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:39.178382 (+ 303us) client_negotiation.cc:769] Sending connection context
0924 16:04:39.178572 (+ 190us) client_negotiation.cc:240] Negotiation successful
0924 16:04:39.178892 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":154,"threads_started":1}
I20260924 16:04:39.179838 8253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:39.161785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:38065 (local address 127.25.109.62:42581)
0924 16:04:39.162190 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:39.162210 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:39.162542 (+ 332us) server_negotiation.cc:407] Connection header received
0924 16:04:39.164043 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:39.164054 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:39.164348 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:39.164736 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:39.166184 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.167614 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.169594 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.170384 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.170573 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:39.173898 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:39.173904 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:39.173921 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:39.173955 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:39.176178 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:39.176846 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:39.176855 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:39.176864 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:39.177040 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:39.177451 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:39.177457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:39.177462 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:39.177942 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:39.178087 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:39.178734 (+ 647us) server_negotiation.cc:299] Negotiation successful
0924 16:04:39.179047 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":111,"threads_started":1}
I20260924 16:04:39.181578 8137 raft_consensus.cc:348] T 7a3ccef5e82641dca568446be08eeca0 P ccf0f95a39c846c9acc0c1c9270f6b5e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63de18a948845c6b7e1dc33a4192d45" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40103 } } peers { permanent_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 42331 } } peers { permanent_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33745 } }
I20260924 16:04:39.182178 8137 raft_consensus.cc:729] T 7a3ccef5e82641dca568446be08eeca0 P ccf0f95a39c846c9acc0c1c9270f6b5e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf0f95a39c846c9acc0c1c9270f6b5e, State: Initialized, Role: FOLLOWER
I20260924 16:04:39.182659 8137 consensus_queue.cc:260] T 7a3ccef5e82641dca568446be08eeca0 P ccf0f95a39c846c9acc0c1c9270f6b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63de18a948845c6b7e1dc33a4192d45" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40103 } } peers { permanent_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 42331 } } peers { permanent_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33745 } }
I20260924 16:04:39.183211 8251 heartbeater.cc:347] Connected to a master server at 127.25.109.62:42581
I20260924 16:04:39.183446 8251 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:39.183727 8137 ts_tablet_manager.cc:1428] T 7a3ccef5e82641dca568446be08eeca0 P ccf0f95a39c846c9acc0c1c9270f6b5e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 16:04:39.183955 8251 heartbeater.cc:511] Master 127.25.109.62:42581 requested a full tablet report, sending...
I20260924 16:04:39.186508 8054 ts_manager.cc:194] Registered new tserver with Master: ccf0f95a39c846c9acc0c1c9270f6b5e (127.25.109.1:33745)
I20260924 16:04:39.190408 8054 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.1@KRBTEST.COM'} at 127.25.109.1:38065
I20260924 16:04:39.193058 8251 heartbeater.cc:503] Master 127.25.109.62:42581 was elected leader, sending a full tablet report...
Sep 24 16:04:39 dist-test-slave-n106 krb5kdc[7459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265878, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for kudu/127.25.109.1@KRBTEST.COM
I20260924 16:04:39.200634 8258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:39.189777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:33745 (local address 127.0.0.1:36838)
0924 16:04:39.190441 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:39.190475 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:39.190636 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:39.191094 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:39.191100 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:39.191518 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:39.191618 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:39.191625 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.192650 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.192656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:39.193339 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:39.193348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.194117 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.194121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:39.194228 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:39.194824 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:39.194835 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:39.196715 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:39.198977 (+ 2262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:39.198985 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:39.198991 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:39.199208 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:39.199511 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:39.199514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:39.199515 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:39.199591 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:39.200066 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:39.200071 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:39.200170 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 16:04:39.200279 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 16:04:39.200486 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":109,"threads_started":1}
I20260924 16:04:39.200728 8259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:39.189827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36838 (local address 127.25.109.1:33745)
0924 16:04:39.190199 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:39.190210 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:39.190745 (+ 535us) server_negotiation.cc:407] Connection header received
0924 16:04:39.190800 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:39.190804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:39.190950 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:39.191092 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:39.191739 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.192541 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.193481 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.193982 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.194095 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:39.196859 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:39.196868 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:39.196881 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:39.196907 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:39.198857 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:39.199317 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:39.199324 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:39.199325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:39.199416 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:39.199674 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:39.199677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:39.199679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:39.199959 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:39.200088 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:39.200382 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 16:04:39.200572 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":91,"threads_started":1}
I20260924 16:04:39.201675 8181 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a3ccef5e82641dca568446be08eeca0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:38 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:36838
I20260924 16:04:39.202523 8260 tablet_replica.cc:330] stopping tablet replica
I20260924 16:04:39.202808 8260 raft_consensus.cc:2229] T 7a3ccef5e82641dca568446be08eeca0 P ccf0f95a39c846c9acc0c1c9270f6b5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:04:39.203051 8260 raft_consensus.cc:2258] T 7a3ccef5e82641dca568446be08eeca0 P ccf0f95a39c846c9acc0c1c9270f6b5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:04:39.204026 8260 ts_tablet_manager.cc:1905] T 7a3ccef5e82641dca568446be08eeca0 P ccf0f95a39c846c9acc0c1c9270f6b5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:04:39.207647 8260 ts_tablet_manager.cc:1918] T 7a3ccef5e82641dca568446be08eeca0 P ccf0f95a39c846c9acc0c1c9270f6b5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:04:39.207736 8260 log.cc:1196] T 7a3ccef5e82641dca568446be08eeca0 P ccf0f95a39c846c9acc0c1c9270f6b5e: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/wal/wals/7a3ccef5e82641dca568446be08eeca0
I20260924 16:04:39.208045 8260 ts_tablet_manager.cc:1939] T 7a3ccef5e82641dca568446be08eeca0 P ccf0f95a39c846c9acc0c1c9270f6b5e: Deleting consensus metadata
I20260924 16:04:39.209821 8039 catalog_manager.cc:4841] TS ccf0f95a39c846c9acc0c1c9270f6b5e (127.25.109.1:33745): tablet 7a3ccef5e82641dca568446be08eeca0 (table test-table [id=841ba3e84ad2405bb2762729129d33dd]) successfully deleted
W20260924 16:04:39.284888 8255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:39.285171 8255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:39.285244 8255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:39.289258 8255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:39.289331 8255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:39.289418 8255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:04:39.294559 8255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:41887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:42331
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=39489
--tserver_master_addrs=127.25.109.62:42581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:04:39.295600 8255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:39.296703 8255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:39.306370 8269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:39.306370 8266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:39.306471 8267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:39.306789 8255 server_base.cc:1034] running on GCE node
I20260924 16:04:39.307211 8255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:39.307822 8255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:39.309051 8255 hybrid_clock.cc:648] HybridClock initialized: now 1790265879309034 us; error 27 us; skew 500 ppm
Sep 24 16:04:39 dist-test-slave-n106 krb5kdc[7459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265879, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:39.311815 8255 init.cc:377] Logged in from keytab as kudu/127.25.109.2@KRBTEST.COM (short username kudu)
I20260924 16:04:39.312686 8255 webserver.cc:460] Webserver started at http://127.25.109.2:39489/ using document root <none> and password file <none>
I20260924 16:04:39.313206 8255 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:39.313278 8255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:39.317216 8255 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:04:39.319453 8276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:39.320729 8255 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:04:39.320823 8255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "7ee64b2ecdbd495295a0dbc199c41f57"
format_stamp: "Formatted at 2026-09-24 16:04:38 on dist-test-slave-n106"
server_key: "76a22351cc78d1e13d487bdeb99bd648"
server_key_iv: "1b1be215adc5f8db2635f225265ee31f"
server_key_version: "encryptionkey@0"
I20260924 16:04:39.321326 8255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:39.333561 8255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:39.337332 8255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:39.337544 8255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:39.338279 8255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:39.339720 8283 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260924 16:04:39.343564 8283 ts_tablet_manager.cc:1852] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 16:04:39.343969 8283 ts_tablet_manager.cc:1905] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:04:39.347743 8283 ts_tablet_manager.cc:1918] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:04:39.347879 8283 log.cc:1196] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/wal/wals/7a3ccef5e82641dca568446be08eeca0
I20260924 16:04:39.348227 8283 ts_tablet_manager.cc:1939] T 7a3ccef5e82641dca568446be08eeca0 P 7ee64b2ecdbd495295a0dbc199c41f57: Deleting consensus metadata
I20260924 16:04:39.348611 8255 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 16:04:39.348675 8255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.000s
I20260924 16:04:39.348728 8255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:39.348750 8255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:39.357861 8255 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:42331
I20260924 16:04:39.357898 8392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:42331 every 8 connection(s)
I20260924 16:04:39.358721 8255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:04:39.360615 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 8255
I20260924 16:04:39.361047 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:40103
--local_ip_for_outbound_sockets=127.25.109.3
--tserver_master_addrs=127.25.109.62:42581
--webserver_port=45117
--webserver_interface=127.25.109.3
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:41887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:04:39 dist-test-slave-n106 krb5kdc[7459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265879, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:04:39.372387 8253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:39.360501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:34101 (local address 127.25.109.62:42581)
0924 16:04:39.360663 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:39.360683 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:39.361661 (+ 978us) server_negotiation.cc:407] Connection header received
0924 16:04:39.362813 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:39.362816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:39.362873 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:39.362956 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:39.364259 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.364866 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.365984 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.366232 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.366345 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:39.368875 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:39.368888 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:39.368890 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:39.368915 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:39.370363 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:39.370867 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:39.370871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:39.370873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:39.370954 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:39.371302 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:39.371306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:39.371308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:39.371489 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:39.371598 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:39.372080 (+ 482us) server_negotiation.cc:299] Negotiation successful
0924 16:04:39.372210 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:04:39.373238 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:39.360989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.2:34101)
0924 16:04:39.361512 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:39.361552 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:39.362592 (+ 1040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:39.363139 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:39.363146 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:39.363543 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:39.364104 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:39.364117 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.364995 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.364998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:39.365854 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:39.365863 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.366380 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.366384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:39.366557 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:39.367216 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:39.367232 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:39.368693 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:39.370485 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:39.370491 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:39.370506 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:39.370761 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:39.371053 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:39.371056 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:39.371058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:39.371204 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:39.371596 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:39.371602 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:39.371849 (+ 247us) client_negotiation.cc:769] Sending connection context
0924 16:04:39.372038 (+ 189us) client_negotiation.cc:240] Negotiation successful
0924 16:04:39.372356 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":123,"threads_started":1}
I20260924 16:04:39.374413 8393 heartbeater.cc:347] Connected to a master server at 127.25.109.62:42581
I20260924 16:04:39.374666 8393 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:39.375139 8393 heartbeater.cc:511] Master 127.25.109.62:42581 requested a full tablet report, sending...
I20260924 16:04:39.376191 8054 ts_manager.cc:194] Registered new tserver with Master: 7ee64b2ecdbd495295a0dbc199c41f57 (127.25.109.2:42331)
I20260924 16:04:39.376993 8054 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.2@KRBTEST.COM'} at 127.25.109.2:34101
W20260924 16:04:39.474954 8396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:39.475221 8396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:39.475266 8396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:39.479312 8396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:39.479372 8396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:39.479432 8396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:04:39.485152 8396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:41887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:40103
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=45117
--tserver_master_addrs=127.25.109.62:42581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:04:39.486186 8396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:39.487195 8396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:39.496791 8402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:39.496903 8401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:39.496795 8404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:39.496877 8396 server_base.cc:1034] running on GCE node
I20260924 16:04:39.497383 8396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:39.498042 8396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:39.499208 8396 hybrid_clock.cc:648] HybridClock initialized: now 1790265879499185 us; error 31 us; skew 500 ppm
Sep 24 16:04:39 dist-test-slave-n106 krb5kdc[7459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265879, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:39.502061 8396 init.cc:377] Logged in from keytab as kudu/127.25.109.3@KRBTEST.COM (short username kudu)
I20260924 16:04:39.503108 8396 webserver.cc:460] Webserver started at http://127.25.109.3:45117/ using document root <none> and password file <none>
I20260924 16:04:39.503705 8396 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:39.503815 8396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:39.507869 8396 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:04:39.509979 8411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:39.511270 8396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:04:39.511410 8396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "d63de18a948845c6b7e1dc33a4192d45"
format_stamp: "Formatted at 2026-09-24 16:04:38 on dist-test-slave-n106"
server_key: "f65498179a7936acc8fcffdf5b21eab0"
server_key_iv: "8f4c22597cf1361ca8ec46f7625cead1"
server_key_version: "encryptionkey@0"
I20260924 16:04:39.511996 8396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:39.524677 8396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:39.528102 8396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:39.528301 8396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:39.528913 8396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:39.529731 8396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:39.529804 8396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:39.529876 8396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:39.529928 8396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:39.538950 8396 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:40103
I20260924 16:04:39.538971 8525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:40103 every 8 connection(s)
I20260924 16:04:39.539968 8396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:04:39.548130 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 8396
Sep 24 16:04:39 dist-test-slave-n106 krb5kdc[7459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265879, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:04:39.554188 8253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:39.541806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:40907 (local address 127.25.109.62:42581)
0924 16:04:39.541962 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:39.541968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:39.542884 (+ 916us) server_negotiation.cc:407] Connection header received
0924 16:04:39.544233 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:39.544236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:39.544284 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:39.544357 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:39.545589 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.546101 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.547200 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.547408 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.547480 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:39.550260 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:39.550264 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:39.550267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:39.550293 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:39.551772 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:39.552283 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:39.552287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:39.552288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:39.552403 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:39.552901 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:39.552903 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:39.552905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:39.553076 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:39.553156 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:39.553832 (+ 676us) server_negotiation.cc:299] Negotiation successful
0924 16:04:39.553938 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:04:39.555141 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:39.542260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.3:40907)
0924 16:04:39.542727 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:39.542771 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:39.544034 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:39.544606 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:39.544613 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:39.544964 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:39.545434 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:39.545447 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.546208 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.546210 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:39.547095 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:39.547105 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.547546 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.547548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:39.547684 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:39.548371 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:39.548387 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:39.550093 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:39.551894 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:39.551899 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:39.551914 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:39.552178 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:39.552579 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:39.552584 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:39.552588 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:39.552788 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:39.553195 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:39.553205 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:39.553534 (+ 329us) client_negotiation.cc:769] Sending connection context
0924 16:04:39.553721 (+ 187us) client_negotiation.cc:240] Negotiation successful
0924 16:04:39.554089 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":112,"threads_started":1}
I20260924 16:04:39.556326 8526 heartbeater.cc:347] Connected to a master server at 127.25.109.62:42581
I20260924 16:04:39.556631 8526 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:39.556867 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:39.548883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42890 (local address 127.25.109.62:42581)
0924 16:04:39.549144 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:39.549149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:39.549274 (+ 125us) server_negotiation.cc:407] Connection header received
0924 16:04:39.549424 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:39.549427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:39.549473 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:39.549569 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:39.550372 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.550891 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.551763 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.551986 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.552094 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:39.553565 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:39.553568 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:39.553569 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:39.553588 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:39.555061 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:39.555567 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:39.555570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:39.555572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:39.555659 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:39.555965 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:39.555968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:39.555970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:39.556154 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:39.556370 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:39.556564 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 16:04:39.556699 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":94,"threads_started":1}
I20260924 16:04:39.557267 8526 heartbeater.cc:511] Master 127.25.109.62:42581 requested a full tablet report, sending...
I20260924 16:04:39.558534 8052 ts_manager.cc:194] Registered new tserver with Master: d63de18a948845c6b7e1dc33a4192d45 (127.25.109.3:40103)
I20260924 16:04:39.559119 8052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.3@KRBTEST.COM'} at 127.25.109.3:40907
I20260924 16:04:39.570454 26036 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:04:39.574743 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:39.571068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42902 (local address 127.25.109.62:42581)
0924 16:04:39.571216 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:39.571223 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:39.571350 (+ 127us) server_negotiation.cc:407] Connection header received
0924 16:04:39.571491 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:39.571493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:39.571593 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:39.571682 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:04:39.571945 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.572562 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.573385 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.573612 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.573704 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:39.573959 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:04:39.574357 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:04:39.574446 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:39.574499 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 16:04:39.574551 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 16:04:39.575400 8052 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:42902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:04:39.578761 8052 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:04:39.586802 8181 tablet_service.cc:1467] Processing CreateTablet for tablet 997993bdad914b508f2875580acd7f2d (DEFAULT_TABLE table=test-table [id=aceff007466843e78919bb4eaedece09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:04:39.587459 8181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 997993bdad914b508f2875580acd7f2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:39.590193 8137 tablet_bootstrap.cc:492] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e: Bootstrap starting.
I20260924 16:04:39.591127 8137 tablet_bootstrap.cc:654] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:39.591243 8258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:39.585683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:42331 (local address 127.0.0.1:42710)
0924 16:04:39.585815 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:39.585831 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:39.585907 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:39.586821 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:39.586824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:39.586844 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:39.586945 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:39.586951 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.588689 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.588692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:39.589530 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:39.589540 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.590791 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.590795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:39.590897 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:39.590950 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 16:04:39.591008 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 16:04:39.591097 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":21}
I20260924 16:04:39.591997 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:39.585892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42710 (local address 127.25.109.2:42331)
0924 16:04:39.586268 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:39.586286 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:39.586309 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:04:39.586391 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:39.586397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:39.586625 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:39.586834 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:39.587045 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.588527 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.589666 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.590656 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.591510 (+ 854us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:39.591661 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:39.591719 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 16:04:39.591820 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":133,"threads_started":1}
I20260924 16:04:39.592314 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:39.585943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:40103 (local address 127.0.0.1:47778)
0924 16:04:39.586320 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:39.586333 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:39.586426 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:39.587219 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:39.587222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:39.587241 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:39.587332 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:39.587338 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.588629 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.588633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:39.590374 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:39.590385 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.591835 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.591841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:39.591950 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:39.591978 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 16:04:39.592064 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 16:04:39.592154 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":828,"thread_start_us":84,"threads_started":1}
I20260924 16:04:39.592357 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:39.586378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47778 (local address 127.25.109.3:40103)
0924 16:04:39.586755 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:39.586767 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:39.586790 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:04:39.586861 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:39.586866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:39.587066 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:39.587248 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:39.587465 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.588481 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.590705 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.591710 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.591812 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:39.591938 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:39.592117 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 16:04:39.592218 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":181,"threads_started":1}
I20260924 16:04:39.592636 8137 tablet_bootstrap.cc:492] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e: No bootstrap required, opened a new log
I20260924 16:04:39.592715 8137 ts_tablet_manager.cc:1397] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 16:04:39.593176 8137 raft_consensus.cc:348] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 42331 } } peers { permanent_uuid: "d63de18a948845c6b7e1dc33a4192d45" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40103 } } peers { permanent_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33745 } }
I20260924 16:04:39.593295 8137 raft_consensus.cc:374] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:39.593333 8137 raft_consensus.cc:729] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf0f95a39c846c9acc0c1c9270f6b5e, State: Initialized, Role: FOLLOWER
I20260924 16:04:39.593457 8137 consensus_queue.cc:260] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 42331 } } peers { permanent_uuid: "d63de18a948845c6b7e1dc33a4192d45" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40103 } } peers { permanent_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33745 } }
I20260924 16:04:39.593724 8137 ts_tablet_manager.cc:1428] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:04:39.593567 8327 tablet_service.cc:1467] Processing CreateTablet for tablet 997993bdad914b508f2875580acd7f2d (DEFAULT_TABLE table=test-table [id=aceff007466843e78919bb4eaedece09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:04:39.594691 8327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 997993bdad914b508f2875580acd7f2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:39.595585 8460 tablet_service.cc:1467] Processing CreateTablet for tablet 997993bdad914b508f2875580acd7f2d (DEFAULT_TABLE table=test-table [id=aceff007466843e78919bb4eaedece09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:04:39.596451 8460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 997993bdad914b508f2875580acd7f2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:39.599355 8283 tablet_bootstrap.cc:492] T 997993bdad914b508f2875580acd7f2d P 7ee64b2ecdbd495295a0dbc199c41f57: Bootstrap starting.
I20260924 16:04:39.601577 8283 tablet_bootstrap.cc:654] T 997993bdad914b508f2875580acd7f2d P 7ee64b2ecdbd495295a0dbc199c41f57: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:39.602459 8283 log.cc:824] T 997993bdad914b508f2875580acd7f2d P 7ee64b2ecdbd495295a0dbc199c41f57: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:39.604182 8540 tablet_bootstrap.cc:492] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45: Bootstrap starting.
I20260924 16:04:39.604547 8283 tablet_bootstrap.cc:492] T 997993bdad914b508f2875580acd7f2d P 7ee64b2ecdbd495295a0dbc199c41f57: No bootstrap required, opened a new log
I20260924 16:04:39.604727 8283 ts_tablet_manager.cc:1397] T 997993bdad914b508f2875580acd7f2d P 7ee64b2ecdbd495295a0dbc199c41f57: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 16:04:39.606374 8540 tablet_bootstrap.cc:654] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:39.607647 8540 log.cc:824] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:39.608745 8283 raft_consensus.cc:348] T 997993bdad914b508f2875580acd7f2d P 7ee64b2ecdbd495295a0dbc199c41f57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 42331 } } peers { permanent_uuid: "d63de18a948845c6b7e1dc33a4192d45" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40103 } } peers { permanent_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33745 } }
I20260924 16:04:39.608999 8283 raft_consensus.cc:374] T 997993bdad914b508f2875580acd7f2d P 7ee64b2ecdbd495295a0dbc199c41f57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:39.609072 8283 raft_consensus.cc:729] T 997993bdad914b508f2875580acd7f2d P 7ee64b2ecdbd495295a0dbc199c41f57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ee64b2ecdbd495295a0dbc199c41f57, State: Initialized, Role: FOLLOWER
I20260924 16:04:39.609514 8283 consensus_queue.cc:260] T 997993bdad914b508f2875580acd7f2d P 7ee64b2ecdbd495295a0dbc199c41f57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 42331 } } peers { permanent_uuid: "d63de18a948845c6b7e1dc33a4192d45" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40103 } } peers { permanent_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33745 } }
I20260924 16:04:39.609835 8540 tablet_bootstrap.cc:492] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45: No bootstrap required, opened a new log
I20260924 16:04:39.610029 8540 ts_tablet_manager.cc:1397] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 16:04:39.610139 8393 heartbeater.cc:503] Master 127.25.109.62:42581 was elected leader, sending a full tablet report...
I20260924 16:04:39.610517 8283 ts_tablet_manager.cc:1428] T 997993bdad914b508f2875580acd7f2d P 7ee64b2ecdbd495295a0dbc199c41f57: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 16:04:39.613389 8540 raft_consensus.cc:348] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 42331 } } peers { permanent_uuid: "d63de18a948845c6b7e1dc33a4192d45" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40103 } } peers { permanent_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33745 } }
I20260924 16:04:39.613632 8540 raft_consensus.cc:374] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:39.613708 8540 raft_consensus.cc:729] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d63de18a948845c6b7e1dc33a4192d45, State: Initialized, Role: FOLLOWER
I20260924 16:04:39.614075 8540 consensus_queue.cc:260] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 42331 } } peers { permanent_uuid: "d63de18a948845c6b7e1dc33a4192d45" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40103 } } peers { permanent_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33745 } }
I20260924 16:04:39.614661 8526 heartbeater.cc:503] Master 127.25.109.62:42581 was elected leader, sending a full tablet report...
I20260924 16:04:39.614988 8540 ts_tablet_manager.cc:1428] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260924 16:04:39.631567 8543 raft_consensus.cc:482] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:04:39.631724 8543 raft_consensus.cc:504] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 42331 } } peers { permanent_uuid: "d63de18a948845c6b7e1dc33a4192d45" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40103 } } peers { permanent_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33745 } }
I20260924 16:04:39.632920 8543 leader_election.cc:288] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ee64b2ecdbd495295a0dbc199c41f57 (127.25.109.2:42331), ccf0f95a39c846c9acc0c1c9270f6b5e (127.25.109.1:33745)
I20260924 16:04:39.637650 8259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:39.633311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:48761 (local address 127.25.109.1:33745)
0924 16:04:39.633458 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:39.633462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:39.633664 (+ 202us) server_negotiation.cc:407] Connection header received
0924 16:04:39.633736 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:39.633739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:39.633801 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:39.633900 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:39.634223 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.634948 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.636299 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.636977 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.637083 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:39.637216 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:39.637443 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 16:04:39.637513 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 16:04:39.637642 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:39.633243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:59613 (local address 127.25.109.2:42331)
0924 16:04:39.633366 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:39.633373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:39.633392 (+ 19us) server_negotiation.cc:407] Connection header received
0924 16:04:39.633506 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:39.633510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:39.633569 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:39.633650 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:39.633966 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.634739 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.636299 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.636915 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.637057 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:39.637110 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:39.637443 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 16:04:39.637514 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 16:04:39.637712 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:39.633113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:42331 (local address 127.25.109.3:59613)
0924 16:04:39.633279 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:39.633296 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:39.633385 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:39.633678 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:39.633683 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:39.633705 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:39.633833 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:39.633841 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.634845 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.634848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:39.636166 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:39.636178 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.637093 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.637097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:39.637248 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:39.637324 (+ 76us) client_negotiation.cc:769] Sending connection context
0924 16:04:39.637411 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 16:04:39.637557 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":129}
I20260924 16:04:39.637791 8544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:39.633134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:33745 (local address 127.25.109.3:48761)
0924 16:04:39.633539 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:39.633553 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:39.633657 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:39.633943 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:39.633947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:39.633971 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:39.634098 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:39.634103 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.635043 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.635046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:39.636187 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:39.636194 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.637093 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.637097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:39.637243 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:39.637324 (+ 81us) client_negotiation.cc:769] Sending connection context
0924 16:04:39.637411 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 16:04:39.637556 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":98,"thread_start_us":112,"threads_started":1}
I20260924 16:04:39.638392 8203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997993bdad914b508f2875580acd7f2d" candidate_uuid: "d63de18a948845c6b7e1dc33a4192d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" is_pre_election: true
I20260924 16:04:39.638403 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997993bdad914b508f2875580acd7f2d" candidate_uuid: "d63de18a948845c6b7e1dc33a4192d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" is_pre_election: true
I20260924 16:04:39.638708 8347 raft_consensus.cc:2454] T 997993bdad914b508f2875580acd7f2d P 7ee64b2ecdbd495295a0dbc199c41f57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d63de18a948845c6b7e1dc33a4192d45 in term 0.
I20260924 16:04:39.638708 8203 raft_consensus.cc:2454] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d63de18a948845c6b7e1dc33a4192d45 in term 0.
I20260924 16:04:39.639197 8415 leader_election.cc:302] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7ee64b2ecdbd495295a0dbc199c41f57, ccf0f95a39c846c9acc0c1c9270f6b5e, d63de18a948845c6b7e1dc33a4192d45; no voters:
I20260924 16:04:39.639467 8543 raft_consensus.cc:2781] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:04:39.639550 8543 raft_consensus.cc:482] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:04:39.639603 8543 raft_consensus.cc:3037] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:39.640738 8543 raft_consensus.cc:504] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 42331 } } peers { permanent_uuid: "d63de18a948845c6b7e1dc33a4192d45" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40103 } } peers { permanent_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33745 } }
I20260924 16:04:39.641063 8543 leader_election.cc:288] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45 [CANDIDATE]: Term 1 election: Requested vote from peers 7ee64b2ecdbd495295a0dbc199c41f57 (127.25.109.2:42331), ccf0f95a39c846c9acc0c1c9270f6b5e (127.25.109.1:33745)
I20260924 16:04:39.641393 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997993bdad914b508f2875580acd7f2d" candidate_uuid: "d63de18a948845c6b7e1dc33a4192d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ee64b2ecdbd495295a0dbc199c41f57"
I20260924 16:04:39.641407 8203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997993bdad914b508f2875580acd7f2d" candidate_uuid: "d63de18a948845c6b7e1dc33a4192d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e"
I20260924 16:04:39.641535 8347 raft_consensus.cc:3037] T 997993bdad914b508f2875580acd7f2d P 7ee64b2ecdbd495295a0dbc199c41f57 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:39.641544 8203 raft_consensus.cc:3037] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:39.642537 8347 raft_consensus.cc:2454] T 997993bdad914b508f2875580acd7f2d P 7ee64b2ecdbd495295a0dbc199c41f57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d63de18a948845c6b7e1dc33a4192d45 in term 1.
I20260924 16:04:39.642544 8203 raft_consensus.cc:2454] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d63de18a948845c6b7e1dc33a4192d45 in term 1.
I20260924 16:04:39.642838 8415 leader_election.cc:302] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ccf0f95a39c846c9acc0c1c9270f6b5e, d63de18a948845c6b7e1dc33a4192d45; no voters:
I20260924 16:04:39.643042 8543 raft_consensus.cc:2781] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:39.643280 8543 raft_consensus.cc:686] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45 [term 1 LEADER]: Becoming Leader. State: Replica: d63de18a948845c6b7e1dc33a4192d45, State: Running, Role: LEADER
I20260924 16:04:39.643610 8543 consensus_queue.cc:237] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 42331 } } peers { permanent_uuid: "d63de18a948845c6b7e1dc33a4192d45" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40103 } } peers { permanent_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33745 } }
I20260924 16:04:39.646791 8052 catalog_manager.cc:5495] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45 reported cstate change: term changed from 0 to 1, leader changed from <none> to d63de18a948845c6b7e1dc33a4192d45 (127.25.109.3). New cstate: current_term: 1 leader_uuid: "d63de18a948845c6b7e1dc33a4192d45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 42331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d63de18a948845c6b7e1dc33a4192d45" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33745 } health_report { overall_health: UNKNOWN } } }
W20260924 16:04:39.660920 8252 tablet.cc:2369] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:04:39.670974 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:39.667394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47794 (local address 127.25.109.3:40103)
0924 16:04:39.667574 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:39.667581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:39.667606 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:04:39.667696 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:39.667698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:39.667749 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:39.667858 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:04:39.668107 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.668630 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.669497 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:39.669717 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:39.669825 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:39.670139 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:04:39.670557 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:04:39.670690 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:39.670751 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 16:04:39.670807 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 16:04:39.676980 8347 raft_consensus.cc:1261] T 997993bdad914b508f2875580acd7f2d P 7ee64b2ecdbd495295a0dbc199c41f57 [term 1 FOLLOWER]: Refusing update from remote peer d63de18a948845c6b7e1dc33a4192d45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:04:39.676978 8203 raft_consensus.cc:1261] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [term 1 FOLLOWER]: Refusing update from remote peer d63de18a948845c6b7e1dc33a4192d45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:04:39.677759 8545 consensus_queue.cc:1035] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:04:39.677963 8543 consensus_queue.cc:1035] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 42331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:04:39.686457 8549 mvcc.cc:205] Tried to move back new op lower bound from 7332929043148959744 to 7332929043022729216. Current Snapshot: MvccSnapshot[applied={T|T < 7332929043148959744 or (T in {7332929043148959744})}]
I20260924 16:04:39.693123 8052 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:42902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:04:39.693308 8052 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:42902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:04:39.695724 8052 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67: Sending DeleteTablet for 3 replicas of tablet 997993bdad914b508f2875580acd7f2d
I20260924 16:04:39.696533 8181 tablet_service.cc:1514] Processing DeleteTablet for tablet 997993bdad914b508f2875580acd7f2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:39 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:36838
I20260924 16:04:39.696619 8327 tablet_service.cc:1514] Processing DeleteTablet for tablet 997993bdad914b508f2875580acd7f2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:39 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:42710
I20260924 16:04:39.696801 8260 tablet_replica.cc:330] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e: stopping tablet replica
I20260924 16:04:39.696921 8260 raft_consensus.cc:2229] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:04:39.697093 8260 raft_consensus.cc:2258] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:04:39.697153 8460 tablet_service.cc:1514] Processing DeleteTablet for tablet 997993bdad914b508f2875580acd7f2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:39 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:47778
I20260924 16:04:39.697606 8559 tablet_replica.cc:330] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45: stopping tablet replica
I20260924 16:04:39.697795 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 8093
I20260924 16:04:39.697913 8559 raft_consensus.cc:2229] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:04:39.698156 8260 ts_tablet_manager.cc:1905] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:04:39.698421 8559 raft_consensus.cc:2258] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:04:39.698464 8558 tablet_replica.cc:330] T 997993bdad914b508f2875580acd7f2d P 7ee64b2ecdbd495295a0dbc199c41f57: stopping tablet replica
I20260924 16:04:39.698824 8558 raft_consensus.cc:2229] T 997993bdad914b508f2875580acd7f2d P 7ee64b2ecdbd495295a0dbc199c41f57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:04:39.699128 8558 raft_consensus.cc:2258] T 997993bdad914b508f2875580acd7f2d P 7ee64b2ecdbd495295a0dbc199c41f57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:04:39.700598 8558 ts_tablet_manager.cc:1905] T 997993bdad914b508f2875580acd7f2d P 7ee64b2ecdbd495295a0dbc199c41f57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:04:39.701117 8559 ts_tablet_manager.cc:1905] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:04:39.703343 8558 ts_tablet_manager.cc:1918] T 997993bdad914b508f2875580acd7f2d P 7ee64b2ecdbd495295a0dbc199c41f57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:04:39.703444 8558 log.cc:1196] T 997993bdad914b508f2875580acd7f2d P 7ee64b2ecdbd495295a0dbc199c41f57: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/wal/wals/997993bdad914b508f2875580acd7f2d
I20260924 16:04:39.703723 8558 ts_tablet_manager.cc:1939] T 997993bdad914b508f2875580acd7f2d P 7ee64b2ecdbd495295a0dbc199c41f57: Deleting consensus metadata
I20260924 16:04:39.704388 8559 ts_tablet_manager.cc:1918] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:04:39.704630 8559 log.cc:1196] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/wal/wals/997993bdad914b508f2875580acd7f2d
I20260924 16:04:39.705039 8559 ts_tablet_manager.cc:1939] T 997993bdad914b508f2875580acd7f2d P d63de18a948845c6b7e1dc33a4192d45: Deleting consensus metadata
I20260924 16:04:39.705123 8037 catalog_manager.cc:4841] TS 7ee64b2ecdbd495295a0dbc199c41f57 (127.25.109.2:42331): tablet 997993bdad914b508f2875580acd7f2d (table test-table [id=aceff007466843e78919bb4eaedece09]) successfully deleted
I20260924 16:04:39.706236 8037 catalog_manager.cc:4841] TS d63de18a948845c6b7e1dc33a4192d45 (127.25.109.3:40103): tablet 997993bdad914b508f2875580acd7f2d (table test-table [id=aceff007466843e78919bb4eaedece09]) successfully deleted
W20260924 16:04:39.707690 8039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.109.1:33745 (error 108)
I20260924 16:04:39.707899 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 8255
I20260924 16:04:39.709235 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:39.708967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:33745 (local address 127.0.0.1:36854)
0924 16:04:39.709094 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:39.709166 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.1:33745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 16:04:39.709419 8039 catalog_manager.cc:4584] TS ccf0f95a39c846c9acc0c1c9270f6b5e (127.25.109.1:33745): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 997993bdad914b508f2875580acd7f2d: Network error: Client connection negotiation failed: client connection to 127.25.109.1:33745: connect: Connection refused (error 111)
I20260924 16:04:39.716279 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 8396
I20260924 16:04:39.722215 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 8019
I20260924 16:04:39.728868 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:42581
--webserver_interface=127.25.109.62
--webserver_port=38041
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:41887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:42581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:39.838546 8561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:39.838863 8561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:39.838941 8561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:39.843262 8561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:04:39.843348 8561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:39.843387 8561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:39.843503 8561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:04:39.843544 8561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:04:39.848791 8561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:41887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:42581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:42581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=38041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:04:39.849850 8561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:39.850863 8561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:39.859382 8566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:39.859395 8569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:39.859393 8567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:39.859444 8561 server_base.cc:1034] running on GCE node
I20260924 16:04:39.860253 8561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:39.861342 8561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:39.862533 8561 hybrid_clock.cc:648] HybridClock initialized: now 1790265879862499 us; error 47 us; skew 500 ppm
Sep 24 16:04:39 dist-test-slave-n106 krb5kdc[7459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265879, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:39.865304 8561 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:04:39.866192 8561 webserver.cc:460] Webserver started at http://127.25.109.62:38041/ using document root <none> and password file <none>
I20260924 16:04:39.866737 8561 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:39.866827 8561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:39.870646 8561 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260924 16:04:39.872797 8576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:39.874094 8561 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:04:39.874235 8561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "38fa2227488d452a92f88ae33c25bf67"
format_stamp: "Formatted at 2026-09-24 16:04:37 on dist-test-slave-n106"
server_key: "17b0a5cdb7457c565552f518246b0f56"
server_key_iv: "43045f6b8bb5329960d699a2ffb73e96"
server_key_version: "encryptionkey@0"
I20260924 16:04:39.874782 8561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:39.899307 8561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:39.902858 8561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:39.903081 8561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:39.910499 8561 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:42581
I20260924 16:04:39.910800 8628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:42581 every 8 connection(s)
I20260924 16:04:39.911675 8561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:04:39.915151 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 8561
I20260924 16:04:39.915717 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:33745
--local_ip_for_outbound_sockets=127.25.109.1
--tserver_master_addrs=127.25.109.62:42581
--webserver_port=44365
--webserver_interface=127.25.109.1
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:41887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:04:39.916601 8629 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:04:39.919026 8629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67: Bootstrap starting.
I20260924 16:04:39.937561 8629 log.cc:824] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:39.948906 8629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:04:39.949301 8629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67: Bootstrap complete.
I20260924 16:04:39.953426 8629 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fa2227488d452a92f88ae33c25bf67" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 42581 } }
I20260924 16:04:39.953816 8629 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38fa2227488d452a92f88ae33c25bf67, State: Initialized, Role: FOLLOWER
I20260924 16:04:39.954322 8629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fa2227488d452a92f88ae33c25bf67" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 42581 } }
I20260924 16:04:39.954438 8629 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:04:39.954530 8629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:04:39.954630 8629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [term 2 FOLLOWER]: Advancing to term 3
I20260924 16:04:39.956620 8629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fa2227488d452a92f88ae33c25bf67" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 42581 } }
I20260924 16:04:39.956950 8629 leader_election.cc:302] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38fa2227488d452a92f88ae33c25bf67; no voters:
I20260924 16:04:39.957291 8629 leader_election.cc:288] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 16:04:39.957376 8633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 16:04:39.957577 8633 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [term 3 LEADER]: Becoming Leader. State: Replica: 38fa2227488d452a92f88ae33c25bf67, State: Running, Role: LEADER
I20260924 16:04:39.957865 8633 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fa2227488d452a92f88ae33c25bf67" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 42581 } }
I20260924 16:04:39.958530 8629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:04:39.958793 8633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38fa2227488d452a92f88ae33c25bf67. Latest consensus state: current_term: 3 leader_uuid: "38fa2227488d452a92f88ae33c25bf67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fa2227488d452a92f88ae33c25bf67" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 42581 } } }
I20260924 16:04:39.958912 8633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:39.959249 8637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:04:39.959283 8634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "38fa2227488d452a92f88ae33c25bf67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fa2227488d452a92f88ae33c25bf67" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 42581 } } }
I20260924 16:04:39.959491 8634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38fa2227488d452a92f88ae33c25bf67 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:39.965734 8637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:04:39.966419 8637 catalog_manager.cc:1259] Loaded cluster ID: d5a4fc66d2284c1299703c4863799ad0
I20260924 16:04:39.966488 8637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 16:04:39.967635 8637 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f78e46f79ad google::LogMessage::Fail() at ??:0
@ 0x7f78e46fa88a google::LogMessage::SendToLog() at ??:0
@ 0x7f78e46f74de google::LogMessage::Flush() at ??:0
@ 0x7f78e46f8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f78ef03afdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f78ef03bd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f78ef038579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f78ef07c640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f78e69493d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f78e6949c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f78e694b3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f78e693a512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f78e7f7e6db start_thread at ??:0
@ 0x7f78e31f561f clone at ??:0
W20260924 16:04:40.045869 8631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:40.046162 8631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:40.046267 8631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:40.050365 8631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:40.050446 8631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:40.050554 8631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:04:40.055897 8631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:41887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:33745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=44365
--tserver_master_addrs=127.25.109.62:42581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:04:40.057215 8631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:40.058274 8631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:40.069339 8671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:40.069334 8670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:40.069347 8673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:40.069849 8631 server_base.cc:1034] running on GCE node
I20260924 16:04:40.070246 8631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:40.070900 8631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:40.072093 8631 hybrid_clock.cc:648] HybridClock initialized: now 1790265880072061 us; error 51 us; skew 500 ppm
Sep 24 16:04:40 dist-test-slave-n106 krb5kdc[7459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265880, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:40.075064 8631 init.cc:377] Logged in from keytab as kudu/127.25.109.1@KRBTEST.COM (short username kudu)
I20260924 16:04:40.076057 8631 webserver.cc:460] Webserver started at http://127.25.109.1:44365/ using document root <none> and password file <none>
I20260924 16:04:40.076718 8631 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:40.076810 8631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:40.080819 8631 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 16:04:40.083148 8681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:40.084538 8631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:04:40.084666 8631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e"
format_stamp: "Formatted at 2026-09-24 16:04:38 on dist-test-slave-n106"
server_key: "4017e7eca347eb3faa717f9498fc91af"
server_key_iv: "638ec19e00271cde59e6e735835257f0"
server_key_version: "encryptionkey@0"
I20260924 16:04:40.085213 8631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:40.096508 8631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:40.099756 8631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:40.099978 8631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:40.100730 8631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:40.101914 8688 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 16:04:40.106204 8631 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 16:04:40.106268 8631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260924 16:04:40.106320 8631 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 16:04:40.108021 8631 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 16:04:40.108078 8631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 16:04:40.108342 8688 tablet_bootstrap.cc:492] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e: Bootstrap starting.
I20260924 16:04:40.115188 8688 log.cc:824] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:40.118280 8631 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:33745
I20260924 16:04:40.119087 8796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:33745 every 8 connection(s)
I20260924 16:04:40.119290 8631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:04:40.123387 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 8631
I20260924 16:04:40.123894 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:42331
--local_ip_for_outbound_sockets=127.25.109.2
--tserver_master_addrs=127.25.109.62:42581
--webserver_port=39489
--webserver_interface=127.25.109.2
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:41887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:04:40.124763 8688 tablet_bootstrap.cc:492] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:04:40.125202 8688 tablet_bootstrap.cc:492] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e: Bootstrap complete.
I20260924 16:04:40.125605 8688 ts_tablet_manager.cc:1397] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.007s
I20260924 16:04:40.128829 8688 raft_consensus.cc:348] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 42331 } } peers { permanent_uuid: "d63de18a948845c6b7e1dc33a4192d45" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40103 } } peers { permanent_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33745 } }
I20260924 16:04:40.129526 8688 raft_consensus.cc:729] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf0f95a39c846c9acc0c1c9270f6b5e, State: Initialized, Role: FOLLOWER
I20260924 16:04:40.130126 8688 consensus_queue.cc:260] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 42331 } } peers { permanent_uuid: "d63de18a948845c6b7e1dc33a4192d45" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40103 } } peers { permanent_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33745 } }
I20260924 16:04:40.131395 8688 ts_tablet_manager.cc:1428] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260924 16:04:40.193226 8799 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:04:40.121816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.1:41975)
0924 16:04:40.122465 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:40.122538 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:40.123839 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:40.192394 (+ 68555us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":156,"threads_started":1}
W20260924 16:04:40.193986 8797 heartbeater.cc:644] Failed to heartbeat to 127.25.109.62:42581 (0 consecutive failures): Network error: Failed to ping master at 127.25.109.62:42581: Client connection negotiation failed: client connection to 127.25.109.62:42581: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 16:04:40.194797 8799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:40.194528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.1:41283)
0924 16:04:40.194658 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:40.194735 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 16:04:40.195520 8799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:40.195300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.1:42501)
0924 16:04:40.195405 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:40.195448 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 16:04:40.195761 8797 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 16:04:40.241498 8800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:40.241771 8800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:40.241816 8800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:40.246022 8800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:40.246084 8800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:40.246196 8800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:04:40.252506 8800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:41887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:42331
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=39489
--tserver_master_addrs=127.25.109.62:42581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:04:40.253672 8800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:40.254748 8800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:40.264545 8809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:40.264592 8806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:40.264866 8800 server_base.cc:1034] running on GCE node
W20260924 16:04:40.264590 8807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:40.265329 8800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:40.266001 8800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:40.267190 8800 hybrid_clock.cc:648] HybridClock initialized: now 1790265880267169 us; error 33 us; skew 500 ppm
Sep 24 16:04:40 dist-test-slave-n106 krb5kdc[7459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265880, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:40.269965 8800 init.cc:377] Logged in from keytab as kudu/127.25.109.2@KRBTEST.COM (short username kudu)
I20260924 16:04:40.270841 8800 webserver.cc:460] Webserver started at http://127.25.109.2:39489/ using document root <none> and password file <none>
I20260924 16:04:40.271430 8800 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:40.271523 8800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:40.275408 8800 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:04:40.277571 8816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:40.278757 8800 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:04:40.278894 8800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "7ee64b2ecdbd495295a0dbc199c41f57"
format_stamp: "Formatted at 2026-09-24 16:04:38 on dist-test-slave-n106"
server_key: "76a22351cc78d1e13d487bdeb99bd648"
server_key_iv: "1b1be215adc5f8db2635f225265ee31f"
server_key_version: "encryptionkey@0"
I20260924 16:04:40.279441 8800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:40.311254 8800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:40.314275 8800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:40.314494 8800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:40.315124 8800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:40.315996 8800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:40.316071 8800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:40.316140 8800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:40.316184 8800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:40.325299 8800 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:42331
I20260924 16:04:40.325327 8930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:42331 every 8 connection(s)
I20260924 16:04:40.326277 8800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:04:40.330230 8933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:40.328500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.2:51697)
0924 16:04:40.328973 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:40.329085 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":112,"threads_started":1}
W20260924 16:04:40.330727 8931 heartbeater.cc:644] Failed to heartbeat to 127.25.109.62:42581 (0 consecutive failures): Network error: Failed to ping master at 127.25.109.62:42581: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
I20260924 16:04:40.331228 8933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:40.331024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.2:34875)
0924 16:04:40.331129 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:40.331174 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 16:04:40.331784 8933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:40.331613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.2:60849)
0924 16:04:40.331700 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:40.331739 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 16:04:40.331941 8931 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 16:04:40.332213 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 8800
I20260924 16:04:40.332659 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:40103
--local_ip_for_outbound_sockets=127.25.109.3
--tserver_master_addrs=127.25.109.62:42581
--webserver_port=45117
--webserver_interface=127.25.109.3
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:41887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:40.446740 8935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:40.447047 8935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:40.447149 8935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:40.451275 8935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:40.451364 8935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:40.451450 8935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:04:40.456707 8935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:41887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:40103
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=45117
--tserver_master_addrs=127.25.109.62:42581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:04:40.457749 8935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:40.458781 8935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:40.468513 8940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:40.468532 8943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:40.468520 8941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:40.468719 8935 server_base.cc:1034] running on GCE node
I20260924 16:04:40.469211 8935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:40.469889 8935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:40.471095 8935 hybrid_clock.cc:648] HybridClock initialized: now 1790265880471072 us; error 31 us; skew 500 ppm
Sep 24 16:04:40 dist-test-slave-n106 krb5kdc[7459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265880, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:40.474035 8935 init.cc:377] Logged in from keytab as kudu/127.25.109.3@KRBTEST.COM (short username kudu)
I20260924 16:04:40.474963 8935 webserver.cc:460] Webserver started at http://127.25.109.3:45117/ using document root <none> and password file <none>
I20260924 16:04:40.475544 8935 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:40.475638 8935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:40.479578 8935 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:04:40.481726 8950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:40.483088 8935 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:04:40.483210 8935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "d63de18a948845c6b7e1dc33a4192d45"
format_stamp: "Formatted at 2026-09-24 16:04:38 on dist-test-slave-n106"
server_key: "f65498179a7936acc8fcffdf5b21eab0"
server_key_iv: "8f4c22597cf1361ca8ec46f7625cead1"
server_key_version: "encryptionkey@0"
I20260924 16:04:40.483772 8935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:40.494683 8935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:40.497519 8935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:40.497701 8935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:40.498262 8935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:40.499050 8935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:40.499096 8935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:40.499158 8935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:40.499195 8935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:40.507793 8935 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:40103
I20260924 16:04:40.507869 9064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:40103 every 8 connection(s)
I20260924 16:04:40.508837 8935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:04:40.512497 9067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:40.510904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.3:44099)
0924 16:04:40.511353 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:40.511798 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":104,"threads_started":1}
W20260924 16:04:40.512950 9065 heartbeater.cc:644] Failed to heartbeat to 127.25.109.62:42581 (0 consecutive failures): Network error: Failed to ping master at 127.25.109.62:42581: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
I20260924 16:04:40.513489 9067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:40.513293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.3:41879)
0924 16:04:40.513393 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:40.513438 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 16:04:40.514097 9067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:40.513918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.3:52947)
0924 16:04:40.514011 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:40.514051 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 16:04:40.514303 9065 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 16:04:40.518541 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 8935
I20260924 16:04:41.196825 9069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:41.196355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.1:56917)
0924 16:04:41.196670 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:41.196726 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":132,"threads_started":1}
I20260924 16:04:41.332851 9070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:41.332453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.2:58345)
0924 16:04:41.332699 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:41.332752 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":72,"threads_started":1}
I20260924 16:04:41.515296 9071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:41.514848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.3:36573)
0924 16:04:41.515125 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:41.515185 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":77,"threads_started":1}
I20260924 16:04:41.675182 9072 raft_consensus.cc:482] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:04:41.675418 9072 raft_consensus.cc:504] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 42331 } } peers { permanent_uuid: "d63de18a948845c6b7e1dc33a4192d45" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40103 } } peers { permanent_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33745 } }
I20260924 16:04:41.676620 9072 leader_election.cc:288] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7ee64b2ecdbd495295a0dbc199c41f57 (127.25.109.2:42331), d63de18a948845c6b7e1dc33a4192d45 (127.25.109.3:40103)
Sep 24 16:04:41 dist-test-slave-n106 krb5kdc[7459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265880, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for kudu/127.25.109.3@KRBTEST.COM
Sep 24 16:04:41 dist-test-slave-n106 krb5kdc[7459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265880, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for kudu/127.25.109.2@KRBTEST.COM
I20260924 16:04:41.694846 9075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:41.677234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:42791 (local address 127.25.109.3:40103)
0924 16:04:41.677629 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:41.677658 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:41.677712 (+ 54us) server_negotiation.cc:407] Connection header received
0924 16:04:41.678636 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:41.678646 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:41.679118 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:41.679460 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:41.681422 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:41.683016 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:41.685073 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:41.685820 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:41.685994 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:41.689480 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:41.689489 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:41.689527 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:41.689562 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:41.691660 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:41.692307 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:41.692312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:41.692317 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:41.692527 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:41.692958 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:41.692962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:41.692964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:41.693482 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:41.693625 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:41.694356 (+ 731us) server_negotiation.cc:299] Negotiation successful
0924 16:04:41.694658 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":81,"threads_started":1}
I20260924 16:04:41.694846 9073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:41.676984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:39179 (local address 127.25.109.2:42331)
0924 16:04:41.677679 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:41.677698 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:41.677728 (+ 30us) server_negotiation.cc:407] Connection header received
0924 16:04:41.678647 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:41.678653 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:41.679118 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:41.679459 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:41.681422 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:41.683008 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:41.685073 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:41.685820 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:41.685994 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:41.689855 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:41.689865 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:41.689883 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:41.689919 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:41.691892 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:41.692437 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:41.692445 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:41.692450 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:41.692605 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:41.692958 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:41.692962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:41.692964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:41.693482 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:41.693625 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:41.694355 (+ 730us) server_negotiation.cc:299] Negotiation successful
0924 16:04:41.694658 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":119,"threads_started":1}
I20260924 16:04:41.694840 9074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:41.677061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:40103 (local address 127.25.109.1:42791)
0924 16:04:41.677346 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:41.677359 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:41.677449 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:41.679603 (+ 2154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:41.679610 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:41.680238 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:41.681229 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:41.681253 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:41.683191 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:41.683195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:41.684932 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:41.684945 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:41.685969 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:41.685973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:41.686309 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:41.687226 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:41.687255 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:41.689237 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:41.691793 (+ 2556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:41.691798 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:41.691810 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:41.692184 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:41.692636 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:41.692639 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:41.692640 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:41.692848 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:41.693607 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:41.693614 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:41.693982 (+ 368us) client_negotiation.cc:769] Sending connection context
0924 16:04:41.694194 (+ 212us) client_negotiation.cc:240] Negotiation successful
0924 16:04:41.694582 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":395,"spinlock_wait_cycles":24576,"thread_start_us":72,"threads_started":1}
I20260924 16:04:41.694840 9069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:41.676804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:42331 (local address 127.25.109.1:39179)
0924 16:04:41.676904 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:41.676921 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:41.677028 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:41.679603 (+ 2575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:41.679610 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:41.680238 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:41.681229 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:41.681253 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:41.683191 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:41.683195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:41.684932 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:41.684945 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:41.685969 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:41.685973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:41.686306 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:41.687226 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:41.687255 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:41.689667 (+ 2412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:41.692002 (+ 2335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:41.692008 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:41.692031 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:41.692285 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:41.692693 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:41.692699 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:41.692702 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:41.692848 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:41.693607 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:41.693614 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:41.693982 (+ 368us) client_negotiation.cc:769] Sending connection context
0924 16:04:41.694194 (+ 212us) client_negotiation.cc:240] Negotiation successful
0924 16:04:41.694632 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":731,"spinlock_wait_cycles":46336}
W20260924 16:04:41.697332 8682 leader_election.cc:341] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ee64b2ecdbd495295a0dbc199c41f57 (127.25.109.2:42331): Not found: Tablet not found: 997993bdad914b508f2875580acd7f2d
W20260924 16:04:41.697556 8682 leader_election.cc:341] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d63de18a948845c6b7e1dc33a4192d45 (127.25.109.3:40103): Not found: Tablet not found: 997993bdad914b508f2875580acd7f2d
I20260924 16:04:41.697614 8682 leader_election.cc:302] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ccf0f95a39c846c9acc0c1c9270f6b5e; no voters: 7ee64b2ecdbd495295a0dbc199c41f57, d63de18a948845c6b7e1dc33a4192d45
I20260924 16:04:41.697921 9072 raft_consensus.cc:2726] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:04:42.197952 9076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:42.197567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.1:47031)
0924 16:04:42.197798 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:42.197855 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":75,"threads_started":1}
I20260924 16:04:42.333853 9077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:42.333501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.2:60063)
0924 16:04:42.333719 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:42.333770 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":68,"threads_started":1}
I20260924 16:04:42.516376 9078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:42.515988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.3:43921)
0924 16:04:42.516232 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:42.516284 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":68,"threads_started":1}
I20260924 16:04:43.199343 9079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:43.198803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.1:41513)
0924 16:04:43.199174 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:43.199240 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":75,"threads_started":1}
I20260924 16:04:43.335093 9080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:43.334673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.2:56705)
0924 16:04:43.334938 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:43.334997 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":69,"threads_started":1}
I20260924 16:04:43.491582 9081 raft_consensus.cc:482] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:04:43.491743 9081 raft_consensus.cc:504] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 42331 } } peers { permanent_uuid: "d63de18a948845c6b7e1dc33a4192d45" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40103 } } peers { permanent_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33745 } }
I20260924 16:04:43.492166 9081 leader_election.cc:288] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7ee64b2ecdbd495295a0dbc199c41f57 (127.25.109.2:42331), d63de18a948845c6b7e1dc33a4192d45 (127.25.109.3:40103)
W20260924 16:04:43.493094 8682 leader_election.cc:341] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ee64b2ecdbd495295a0dbc199c41f57 (127.25.109.2:42331): Not found: Tablet not found: 997993bdad914b508f2875580acd7f2d
W20260924 16:04:43.493237 8682 leader_election.cc:341] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d63de18a948845c6b7e1dc33a4192d45 (127.25.109.3:40103): Not found: Tablet not found: 997993bdad914b508f2875580acd7f2d
I20260924 16:04:43.493296 8682 leader_election.cc:302] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ccf0f95a39c846c9acc0c1c9270f6b5e; no voters: 7ee64b2ecdbd495295a0dbc199c41f57, d63de18a948845c6b7e1dc33a4192d45
I20260924 16:04:43.493495 9081 raft_consensus.cc:2726] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:04:43.517725 9082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:43.517256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.3:48059)
0924 16:04:43.517553 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:43.517609 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":84,"threads_started":1}
I20260924 16:04:44.200585 9083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:44.200118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.1:49245)
0924 16:04:44.200385 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:44.200485 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":97,"threads_started":1}
I20260924 16:04:44.336274 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:44.335906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.2:55707)
0924 16:04:44.336134 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:44.336188 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":70,"threads_started":1}
I20260924 16:04:44.518848 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:44.518413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.3:38979)
0924 16:04:44.518692 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:44.518749 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":71,"threads_started":1}
I20260924 16:04:45.201965 9086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:45.201477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.1:41655)
0924 16:04:45.201804 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:45.201862 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":101,"threads_started":1}
I20260924 16:04:45.337493 9087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:45.337020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.2:56581)
0924 16:04:45.337315 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:45.337385 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":102,"threads_started":1}
I20260924 16:04:45.520254 9088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:45.519719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.3:50275)
0924 16:04:45.520026 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:45.520137 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":100,"threads_started":1}
I20260924 16:04:46.203202 9089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:46.202743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.1:35993)
0924 16:04:46.203030 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:46.203099 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":77,"threads_started":1}
I20260924 16:04:46.338675 9090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:46.338251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.2:59521)
0924 16:04:46.338512 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:46.338572 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":90,"threads_started":1}
I20260924 16:04:46.441845 9091 raft_consensus.cc:482] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:04:46.442037 9091 raft_consensus.cc:504] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 42331 } } peers { permanent_uuid: "d63de18a948845c6b7e1dc33a4192d45" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40103 } } peers { permanent_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33745 } }
I20260924 16:04:46.442515 9091 leader_election.cc:288] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7ee64b2ecdbd495295a0dbc199c41f57 (127.25.109.2:42331), d63de18a948845c6b7e1dc33a4192d45 (127.25.109.3:40103)
W20260924 16:04:46.443269 8682 leader_election.cc:341] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ee64b2ecdbd495295a0dbc199c41f57 (127.25.109.2:42331): Not found: Tablet not found: 997993bdad914b508f2875580acd7f2d
W20260924 16:04:46.443387 8682 leader_election.cc:341] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d63de18a948845c6b7e1dc33a4192d45 (127.25.109.3:40103): Not found: Tablet not found: 997993bdad914b508f2875580acd7f2d
I20260924 16:04:46.443439 8682 leader_election.cc:302] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ccf0f95a39c846c9acc0c1c9270f6b5e; no voters: 7ee64b2ecdbd495295a0dbc199c41f57, d63de18a948845c6b7e1dc33a4192d45
I20260924 16:04:46.443578 9091 raft_consensus.cc:2726] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:04:46.521562 9092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:46.521106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.3:44777)
0924 16:04:46.521401 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:46.521457 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":97,"threads_started":1}
I20260924 16:04:47.204535 9093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:47.204037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.1:48225)
0924 16:04:47.204310 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:47.204369 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":94,"threads_started":1}
I20260924 16:04:47.339962 9094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:47.339435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.2:43399)
0924 16:04:47.339735 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:47.339829 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":106,"threads_started":1}
I20260924 16:04:47.523015 9095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:47.522501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.3:55495)
0924 16:04:47.522847 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:47.522915 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":93,"threads_started":1}
I20260924 16:04:48.205996 9096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:48.205521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.1:53703)
0924 16:04:48.205825 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:48.205893 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":88,"threads_started":1}
I20260924 16:04:48.341327 9097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:48.340857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.2:57911)
0924 16:04:48.341154 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:48.341222 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":79,"threads_started":1}
I20260924 16:04:48.524497 9098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:48.523938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.3:34207)
0924 16:04:48.524245 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:48.524311 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":76,"threads_started":1}
I20260924 16:04:49.207319 9099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:49.206869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.1:55167)
0924 16:04:49.207145 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:49.207213 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":79,"threads_started":1}
I20260924 16:04:49.342548 9100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:49.342144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.2:57003)
0924 16:04:49.342392 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:49.342450 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":76,"threads_started":1}
I20260924 16:04:49.525761 9101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:49.525337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.3:42419)
0924 16:04:49.525601 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:49.525660 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
I20260924 16:04:50.208865 9102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:50.208325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.1:55041)
0924 16:04:50.208696 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:50.208763 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":173,"threads_started":1}
I20260924 16:04:50.343698 9103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:50.343249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.2:33069)
0924 16:04:50.343509 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:50.343564 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":86,"threads_started":1}
I20260924 16:04:50.527119 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:50.526650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.3:59119)
0924 16:04:50.526947 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:50.527013 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":94,"threads_started":1}
I20260924 16:04:50.737879 9105 raft_consensus.cc:482] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:04:50.738027 9105 raft_consensus.cc:504] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 42331 } } peers { permanent_uuid: "d63de18a948845c6b7e1dc33a4192d45" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40103 } } peers { permanent_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33745 } }
I20260924 16:04:50.738426 9105 leader_election.cc:288] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7ee64b2ecdbd495295a0dbc199c41f57 (127.25.109.2:42331), d63de18a948845c6b7e1dc33a4192d45 (127.25.109.3:40103)
W20260924 16:04:50.739442 8682 leader_election.cc:341] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ee64b2ecdbd495295a0dbc199c41f57 (127.25.109.2:42331): Not found: Tablet not found: 997993bdad914b508f2875580acd7f2d
W20260924 16:04:50.739624 8682 leader_election.cc:341] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d63de18a948845c6b7e1dc33a4192d45 (127.25.109.3:40103): Not found: Tablet not found: 997993bdad914b508f2875580acd7f2d
I20260924 16:04:50.739694 8682 leader_election.cc:302] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ccf0f95a39c846c9acc0c1c9270f6b5e; no voters: 7ee64b2ecdbd495295a0dbc199c41f57, d63de18a948845c6b7e1dc33a4192d45
I20260924 16:04:50.739857 9105 raft_consensus.cc:2726] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:04:51.210176 9106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:51.209719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.1:48509)
0924 16:04:51.209992 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:51.210064 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":75,"threads_started":1}
I20260924 16:04:51.344977 9107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:51.344545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.2:54223)
0924 16:04:51.344812 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:51.344875 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":82,"threads_started":1}
I20260924 16:04:51.528545 9108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:51.528005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.3:55327)
0924 16:04:51.528284 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:51.528349 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":94,"threads_started":1}
I20260924 16:04:52.211439 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:52.210991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.1:33767)
0924 16:04:52.211278 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:52.211340 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":82,"threads_started":1}
I20260924 16:04:52.346246 9110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:52.345782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.2:33437)
0924 16:04:52.346072 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:52.346138 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":73,"threads_started":1}
I20260924 16:04:52.529909 9111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:52.529466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.3:41211)
0924 16:04:52.529744 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:52.529807 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":83,"threads_started":1}
I20260924 16:04:52.822348 9112 raft_consensus.cc:482] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:04:52.822496 9112 raft_consensus.cc:504] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ee64b2ecdbd495295a0dbc199c41f57" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 42331 } } peers { permanent_uuid: "d63de18a948845c6b7e1dc33a4192d45" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 40103 } } peers { permanent_uuid: "ccf0f95a39c846c9acc0c1c9270f6b5e" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33745 } }
I20260924 16:04:52.822968 9112 leader_election.cc:288] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7ee64b2ecdbd495295a0dbc199c41f57 (127.25.109.2:42331), d63de18a948845c6b7e1dc33a4192d45 (127.25.109.3:40103)
W20260924 16:04:52.823879 8682 leader_election.cc:341] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ee64b2ecdbd495295a0dbc199c41f57 (127.25.109.2:42331): Not found: Tablet not found: 997993bdad914b508f2875580acd7f2d
W20260924 16:04:52.824023 8682 leader_election.cc:341] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d63de18a948845c6b7e1dc33a4192d45 (127.25.109.3:40103): Not found: Tablet not found: 997993bdad914b508f2875580acd7f2d
I20260924 16:04:52.824064 8682 leader_election.cc:302] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ccf0f95a39c846c9acc0c1c9270f6b5e; no voters: 7ee64b2ecdbd495295a0dbc199c41f57, d63de18a948845c6b7e1dc33a4192d45
I20260924 16:04:52.824218 9112 raft_consensus.cc:2726] T 997993bdad914b508f2875580acd7f2d P ccf0f95a39c846c9acc0c1c9270f6b5e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:04:53.212870 9113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:53.212359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.1:38575)
0924 16:04:53.212699 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:53.212767 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":144,"threads_started":1}
I20260924 16:04:53.347569 9114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:53.347115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.2:39537)
0924 16:04:53.347392 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:53.347460 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":76,"threads_started":1}
I20260924 16:04:53.531144 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:53.530706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.3:33977)
0924 16:04:53.530965 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:53.531026 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":80,"threads_started":1}
I20260924 16:04:54.214262 9116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:54.213773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.1:55491)
0924 16:04:54.214091 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:54.214158 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":85,"threads_started":1}
I20260924 16:04:54.348850 9117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:54.348343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.2:45079)
0924 16:04:54.348682 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:54.348744 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":162,"threads_started":1}
I20260924 16:04:54.532421 9118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:54.531970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.3:40451)
0924 16:04:54.532238 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:54.532300 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":83,"threads_started":1}
I20260924 16:04:55.215839 9119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:55.215392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.1:55243)
0924 16:04:55.215683 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:55.215742 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":104,"threads_started":1}
I20260924 16:04:55.350243 9120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:55.349822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.2:40453)
0924 16:04:55.350083 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:55.350144 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
I20260924 16:04:55.525539 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 8631
I20260924 16:04:55.532160 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 8800
I20260924 16:04:55.534411 9121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:55.533973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:42581 (local address 127.25.109.3:34237)
0924 16:04:55.534260 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:55.534317 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":96,"threads_started":1}
I20260924 16:04:55.538460 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 8935
2026-09-24T16:04:55Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19106 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:04:55 dist-test-slave-n106 krb5kdc[9124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:04:55 dist-test-slave-n106 krb5kdc[9124](info): set up 2 sockets
Sep 24 16:04:55 dist-test-slave-n106 krb5kdc[9124](info): commencing operation
krb5kdc: starting...
W20260924 16:04:56.585819 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:04:56 dist-test-slave-n106 krb5kdc[9124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:04:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:04:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
I20260924 16:04:56.713286 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:40819
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:40819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:56.828598 9137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:56.828882 9137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:56.828924 9137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:56.833026 9137 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 16:04:56.833082 9137 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 16:04:56.833098 9137 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 16:04:56.833113 9137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:04:56.833153 9137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:56.833192 9137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:56.833220 9137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:04:56.833253 9137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:04:56.838721 9137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:40819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:40819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:04:56.839862 9137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:56.841005 9137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:56.850167 9143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:56.850144 9145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:56.850238 9137 server_base.cc:1034] running on GCE node
W20260924 16:04:56.850168 9142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:56.851016 9137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:56.852253 9137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:56.853469 9137 hybrid_clock.cc:648] HybridClock initialized: now 1790265896853446 us; error 38 us; skew 500 ppm
Sep 24 16:04:56 dist-test-slave-n106 krb5kdc[9124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265896, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:56.856345 9137 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:04:56.857390 9137 webserver.cc:460] Webserver started at http://127.25.109.62:35681/ using document root <none> and password file <none>
I20260924 16:04:56.858018 9137 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:56.858119 9137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:56.858376 9137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:56.860682 9137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "2e6988e6593c40fbab8f0625315830d3"
format_stamp: "Formatted at 2026-09-24 16:04:56 on dist-test-slave-n106"
server_key: "e70a4b38b417a5725e3cbbd04d0a5d6c"
server_key_iv: "3a59fa88f803bba63b4e1a9b0f2fa4cc"
server_key_version: "encryptionkey@0"
I20260924 16:04:56.861299 9137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "2e6988e6593c40fbab8f0625315830d3"
format_stamp: "Formatted at 2026-09-24 16:04:56 on dist-test-slave-n106"
server_key: "e70a4b38b417a5725e3cbbd04d0a5d6c"
server_key_iv: "3a59fa88f803bba63b4e1a9b0f2fa4cc"
server_key_version: "encryptionkey@0"
I20260924 16:04:56.865350 9137 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:04:56.867966 9152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:56.869418 9137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:04:56.869572 9137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "2e6988e6593c40fbab8f0625315830d3"
format_stamp: "Formatted at 2026-09-24 16:04:56 on dist-test-slave-n106"
server_key: "e70a4b38b417a5725e3cbbd04d0a5d6c"
server_key_iv: "3a59fa88f803bba63b4e1a9b0f2fa4cc"
server_key_version: "encryptionkey@0"
I20260924 16:04:56.869699 9137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:56.889418 9137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:56.892572 9137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:56.892793 9137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:56.900635 9137 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:40819
I20260924 16:04:56.900938 9205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:40819 every 8 connection(s)
I20260924 16:04:56.901758 9137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:04:56.904846 9206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:56.910229 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 9137
I20260924 16:04:56.910382 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:04:56.910727 26036 external_mini_cluster.cc:1428] Setting key cd2061129e3d8f58741691fa67207746
I20260924 16:04:56.910951 9206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3: Bootstrap starting.
I20260924 16:04:56.913611 9206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:56.914541 9206 log.cc:824] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:56.916754 9206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3: No bootstrap required, opened a new log
Sep 24 16:04:56 dist-test-slave-n106 krb5kdc[9124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:04:56.920637 9206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6988e6593c40fbab8f0625315830d3" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40819 } }
I20260924 16:04:56.920939 9206 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:56.921030 9206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e6988e6593c40fbab8f0625315830d3, State: Initialized, Role: FOLLOWER
I20260924 16:04:56.921594 9206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6988e6593c40fbab8f0625315830d3" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40819 } }
I20260924 16:04:56.921787 9206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:04:56.921885 9206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:04:56.921996 9206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:56.922914 9206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6988e6593c40fbab8f0625315830d3" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40819 } }
I20260924 16:04:56.923283 9206 leader_election.cc:302] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e6988e6593c40fbab8f0625315830d3; no voters:
I20260924 16:04:56.923648 9206 leader_election.cc:288] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:04:56.923772 9211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:56.924008 9211 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [term 1 LEADER]: Becoming Leader. State: Replica: 2e6988e6593c40fbab8f0625315830d3, State: Running, Role: LEADER
I20260924 16:04:56.924455 9211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6988e6593c40fbab8f0625315830d3" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40819 } }
I20260924 16:04:56.925058 9206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:04:56.925921 9211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e6988e6593c40fbab8f0625315830d3. Latest consensus state: current_term: 1 leader_uuid: "2e6988e6593c40fbab8f0625315830d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6988e6593c40fbab8f0625315830d3" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40819 } } }
I20260924 16:04:56.926118 9211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:56.926410 9209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:56.912267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44574 (local address 127.25.109.62:40819)
0924 16:04:56.912801 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:56.912819 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:56.912859 (+ 40us) server_negotiation.cc:407] Connection header received
0924 16:04:56.913546 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:56.913564 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:56.913983 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:56.914286 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:56.915238 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:56.916247 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:56.917249 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:56.917731 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:56.917915 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:56.920443 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:56.920453 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:56.920481 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:56.920509 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:56.922489 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:56.922947 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:56.922953 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:56.922965 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:56.923118 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:56.923427 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:56.923432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:56.923437 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:56.924015 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:56.924869 (+ 854us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:56.925040 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 16:04:56.925345 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":210,"threads_started":1}
I20260924 16:04:56.926371 9213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e6988e6593c40fbab8f0625315830d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6988e6593c40fbab8f0625315830d3" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40819 } } }
I20260924 16:04:56.926502 9213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [sys.catalog]: This master's current role is: LEADER
I20260924 16:04:56.926831 9220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:04:56.930033 9220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:04:56.936972 9220 catalog_manager.cc:1347] Generated new cluster ID: 5562202b9b744fc0b78d12c52e0503ce
I20260924 16:04:56.937059 9220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:04:56.952085 9220 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:04:56.953214 9220 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:04:56.959836 9220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3: Generated new TSK 0
I20260924 16:04:56.960624 9220 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
I20260924 16:04:56.995335 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:0
--local_ip_for_outbound_sockets=127.25.109.1
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:40819
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:57.108891 9232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:57.109138 9232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:57.109179 9232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:57.113224 9232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:57.113283 9232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:57.113340 9232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:04:57.118520 9232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:40819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:04:57.119545 9232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:57.120563 9232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:57.130458 9240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:57.130604 9238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:57.130510 9237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:57.130640 9232 server_base.cc:1034] running on GCE node
I20260924 16:04:57.131192 9232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:57.131812 9232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:57.132979 9232 hybrid_clock.cc:648] HybridClock initialized: now 1790265897132958 us; error 30 us; skew 500 ppm
Sep 24 16:04:57 dist-test-slave-n106 krb5kdc[9124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265897, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:57.135816 9232 init.cc:377] Logged in from keytab as kudu/127.25.109.1@KRBTEST.COM (short username kudu)
I20260924 16:04:57.136785 9232 webserver.cc:460] Webserver started at http://127.25.109.1:44523/ using document root <none> and password file <none>
I20260924 16:04:57.137444 9232 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:57.137519 9232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:57.137697 9232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:57.139758 9232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/data/instance:
uuid: "3ab79c4bfa714ec89007db708ee1e66f"
format_stamp: "Formatted at 2026-09-24 16:04:57 on dist-test-slave-n106"
server_key: "00c0523734fef87fc5f2c91428e91271"
server_key_iv: "9e2d5ce57e0f97f4a8de9c85802e0d5c"
server_key_version: "encryptionkey@0"
I20260924 16:04:57.140266 9232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance:
uuid: "3ab79c4bfa714ec89007db708ee1e66f"
format_stamp: "Formatted at 2026-09-24 16:04:57 on dist-test-slave-n106"
server_key: "00c0523734fef87fc5f2c91428e91271"
server_key_iv: "9e2d5ce57e0f97f4a8de9c85802e0d5c"
server_key_version: "encryptionkey@0"
I20260924 16:04:57.144130 9232 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 16:04:57.146638 9247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:57.147912 9232 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 16:04:57.148054 9232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "3ab79c4bfa714ec89007db708ee1e66f"
format_stamp: "Formatted at 2026-09-24 16:04:57 on dist-test-slave-n106"
server_key: "00c0523734fef87fc5f2c91428e91271"
server_key_iv: "9e2d5ce57e0f97f4a8de9c85802e0d5c"
server_key_version: "encryptionkey@0"
I20260924 16:04:57.148177 9232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:57.165338 9232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:57.168567 9232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:57.168773 9232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:57.169389 9232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:57.170249 9232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:57.170320 9232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:57.170395 9232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:57.170447 9232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:57.179581 9232 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:33851
I20260924 16:04:57.179606 9361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:33851 every 8 connection(s)
I20260924 16:04:57.180538 9232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:04:57.182010 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 9232
I20260924 16:04:57.182152 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance
I20260924 16:04:57.182524 26036 external_mini_cluster.cc:1428] Setting key 2aea781d1ed4d255efd8e33e02c3385b
Sep 24 16:04:57 dist-test-slave-n106 krb5kdc[9124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265897, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.2@KRBTEST.COM" created.
I20260924 16:04:57.195705 9209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:57.182446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:34327 (local address 127.25.109.62:40819)
0924 16:04:57.182588 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:57.182595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:57.183771 (+ 1176us) server_negotiation.cc:407] Connection header received
0924 16:04:57.185274 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:57.185278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:57.185332 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:57.185412 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:57.186601 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.187154 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.188314 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.188646 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.188764 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:57.191732 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:57.191736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:57.191739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:57.191768 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:57.193304 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:57.193878 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:57.193881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:57.193883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:57.193966 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:57.194339 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:57.194342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:57.194344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:57.194670 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:57.194787 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:57.195431 (+ 644us) server_negotiation.cc:299] Negotiation successful
0924 16:04:57.195573 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 16:04:57.196836 9364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:57.182968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.1:34327)
0924 16:04:57.183600 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:57.183657 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:57.185065 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:57.185596 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:57.185603 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:57.185964 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:57.186446 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:57.186459 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.187288 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.187291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:57.188185 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:57.188194 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.188805 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.188809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:57.188990 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:57.189611 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:57.189629 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:57.191564 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:57.193427 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:57.193433 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:57.193449 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:57.193737 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:57.194050 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:57.194053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:57.194055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:57.194187 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:57.194812 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:57.194821 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:57.195159 (+ 338us) client_negotiation.cc:769] Sending connection context
0924 16:04:57.195388 (+ 229us) client_negotiation.cc:240] Negotiation successful
0924 16:04:57.195770 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":150,"threads_started":1}
I20260924 16:04:57.198149 9362 heartbeater.cc:347] Connected to a master server at 127.25.109.62:40819
I20260924 16:04:57.198421 9362 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:57.199074 9362 heartbeater.cc:511] Master 127.25.109.62:40819 requested a full tablet report, sending...
I20260924 16:04:57.201009 9170 ts_manager.cc:194] Registered new tserver with Master: 3ab79c4bfa714ec89007db708ee1e66f (127.25.109.1:33851)
I20260924 16:04:57.202731 9170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.1@KRBTEST.COM'} at 127.25.109.1:34327
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
I20260924 16:04:57.210938 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:0
--local_ip_for_outbound_sockets=127.25.109.2
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:40819
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:57.324946 9367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:57.325220 9367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:57.325263 9367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:57.329445 9367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:57.329510 9367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:57.329569 9367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:04:57.334836 9367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:40819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:04:57.335896 9367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:57.337024 9367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:04:57.346980 9367 server_base.cc:1034] running on GCE node
W20260924 16:04:57.347112 9373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:57.346956 9375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:57.346941 9372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:57.347692 9367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:57.348337 9367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:57.349495 9367 hybrid_clock.cc:648] HybridClock initialized: now 1790265897349463 us; error 45 us; skew 500 ppm
Sep 24 16:04:57 dist-test-slave-n106 krb5kdc[9124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265897, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:57.352290 9367 init.cc:377] Logged in from keytab as kudu/127.25.109.2@KRBTEST.COM (short username kudu)
I20260924 16:04:57.353180 9367 webserver.cc:460] Webserver started at http://127.25.109.2:37343/ using document root <none> and password file <none>
I20260924 16:04:57.353767 9367 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:57.353856 9367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:57.354094 9367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:57.356143 9367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/data/instance:
uuid: "11f08c0f2df24d5388cedd0f60a66f92"
format_stamp: "Formatted at 2026-09-24 16:04:57 on dist-test-slave-n106"
server_key: "6f297cfe1d5d9fe15aafc2974ebc4387"
server_key_iv: "59c983a953a0137da7154db4c6f280fe"
server_key_version: "encryptionkey@0"
I20260924 16:04:57.356797 9367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance:
uuid: "11f08c0f2df24d5388cedd0f60a66f92"
format_stamp: "Formatted at 2026-09-24 16:04:57 on dist-test-slave-n106"
server_key: "6f297cfe1d5d9fe15aafc2974ebc4387"
server_key_iv: "59c983a953a0137da7154db4c6f280fe"
server_key_version: "encryptionkey@0"
I20260924 16:04:57.360566 9367 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 16:04:57.363059 9382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:57.364283 9367 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:04:57.364490 9367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "11f08c0f2df24d5388cedd0f60a66f92"
format_stamp: "Formatted at 2026-09-24 16:04:57 on dist-test-slave-n106"
server_key: "6f297cfe1d5d9fe15aafc2974ebc4387"
server_key_iv: "59c983a953a0137da7154db4c6f280fe"
server_key_version: "encryptionkey@0"
I20260924 16:04:57.364612 9367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:57.382493 9367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:57.385814 9367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:57.386034 9367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:57.386682 9367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:57.387633 9367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:57.387707 9367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:57.387784 9367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:57.387826 9367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:57.397194 9367 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:38223
I20260924 16:04:57.397233 9496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:38223 every 8 connection(s)
I20260924 16:04:57.398177 9367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:04:57.407689 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 9367
I20260924 16:04:57.407842 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance
Sep 24 16:04:57 dist-test-slave-n106 krb5kdc[9124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265897, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:04:57.408207 26036 external_mini_cluster.cc:1428] Setting key 450356d43777b5cb7085e8bd649669ad
I20260924 16:04:57.412648 9209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:57.400154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:43769 (local address 127.25.109.62:40819)
0924 16:04:57.400339 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:57.400344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:57.401337 (+ 993us) server_negotiation.cc:407] Connection header received
0924 16:04:57.402699 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:57.402702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:57.402753 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:57.402830 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:57.404109 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.404664 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.405768 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.405981 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.406065 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:57.408781 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:57.408784 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:57.408786 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:57.408809 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:57.410278 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:57.410914 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:57.410917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:57.410918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:57.410993 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:57.411398 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:57.411404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:57.411407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:57.411673 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:57.411778 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:57.412269 (+ 491us) server_negotiation.cc:299] Negotiation successful
0924 16:04:57.412450 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 16:04:57.413407 9499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:57.400645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.2:43769)
0924 16:04:57.401165 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:57.401210 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:57.402459 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:57.403015 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:57.403022 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:57.403386 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:57.403965 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:57.403982 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.404777 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.404780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:57.405664 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:57.405673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.406108 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.406110 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:57.406247 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:57.406798 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:57.406814 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:57.408587 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:57.410401 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:57.410411 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:57.410431 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:57.410810 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:57.411086 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:57.411092 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:57.411097 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:57.411292 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:57.411784 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:57.411789 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:57.412021 (+ 232us) client_negotiation.cc:769] Sending connection context
0924 16:04:57.412189 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 16:04:57.412507 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":115,"threads_started":1}
I20260924 16:04:57.414603 9497 heartbeater.cc:347] Connected to a master server at 127.25.109.62:40819
I20260924 16:04:57.414845 9497 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:57.415337 9497 heartbeater.cc:511] Master 127.25.109.62:40819 requested a full tablet report, sending...
I20260924 16:04:57.416481 9170 ts_manager.cc:194] Registered new tserver with Master: 11f08c0f2df24d5388cedd0f60a66f92 (127.25.109.2:38223)
I20260924 16:04:57.417311 9170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.2@KRBTEST.COM'} at 127.25.109.2:43769
WARNING: no policy specified for kudu/127.25.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
I20260924 16:04:57.435607 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:0
--local_ip_for_outbound_sockets=127.25.109.3
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:40819
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:04:57.549207 9502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:04:57.549525 9502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:04:57.549639 9502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:04:57.553856 9502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:04:57.553942 9502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:04:57.554028 9502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:04:57.559404 9502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:40819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:04:57.560642 9502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:04:57.561769 9502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:04:57.571841 9510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:57.571985 9502 server_base.cc:1034] running on GCE node
W20260924 16:04:57.571902 9508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:04:57.571926 9507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:04:57.572551 9502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:04:57.573256 9502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:04:57.574402 9502 hybrid_clock.cc:648] HybridClock initialized: now 1790265897574385 us; error 28 us; skew 500 ppm
Sep 24 16:04:57 dist-test-slave-n106 krb5kdc[9124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265897, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:04:57.577275 9502 init.cc:377] Logged in from keytab as kudu/127.25.109.3@KRBTEST.COM (short username kudu)
I20260924 16:04:57.578109 9502 webserver.cc:460] Webserver started at http://127.25.109.3:34043/ using document root <none> and password file <none>
I20260924 16:04:57.578644 9502 fs_manager.cc:359] Metadata directory not provided
I20260924 16:04:57.578701 9502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:04:57.578862 9502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:04:57.580951 9502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/data/instance:
uuid: "3c780642dde34cc9b244869a25223c8b"
format_stamp: "Formatted at 2026-09-24 16:04:57 on dist-test-slave-n106"
server_key: "7d8b653ade785dfeeaac4f05d9bcbac3"
server_key_iv: "39f7befb2b91d0ce6f5bcab435f4e500"
server_key_version: "encryptionkey@0"
I20260924 16:04:57.581496 9502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance:
uuid: "3c780642dde34cc9b244869a25223c8b"
format_stamp: "Formatted at 2026-09-24 16:04:57 on dist-test-slave-n106"
server_key: "7d8b653ade785dfeeaac4f05d9bcbac3"
server_key_iv: "39f7befb2b91d0ce6f5bcab435f4e500"
server_key_version: "encryptionkey@0"
I20260924 16:04:57.585188 9502 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 16:04:57.587544 9517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:57.588896 9502 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260924 16:04:57.589035 9502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "3c780642dde34cc9b244869a25223c8b"
format_stamp: "Formatted at 2026-09-24 16:04:57 on dist-test-slave-n106"
server_key: "7d8b653ade785dfeeaac4f05d9bcbac3"
server_key_iv: "39f7befb2b91d0ce6f5bcab435f4e500"
server_key_version: "encryptionkey@0"
I20260924 16:04:57.589154 9502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:04:57.599874 9502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:04:57.603246 9502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:04:57.603467 9502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:04:57.604076 9502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:04:57.605016 9502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:04:57.605089 9502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:57.605165 9502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:04:57.605209 9502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:04:57.614305 9502 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:35851
I20260924 16:04:57.614336 9631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:35851 every 8 connection(s)
I20260924 16:04:57.615227 9502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:04:57.621981 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 9502
I20260924 16:04:57.622128 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance
I20260924 16:04:57.622453 26036 external_mini_cluster.cc:1428] Setting key 57a14f10f45277d4c086652ff39690e9
Sep 24 16:04:57 dist-test-slave-n106 krb5kdc[9124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265897, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:04:57.629407 9209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:57.617112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:45381 (local address 127.25.109.62:40819)
0924 16:04:57.617293 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:57.617298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:57.618131 (+ 833us) server_negotiation.cc:407] Connection header received
0924 16:04:57.619443 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:57.619446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:57.619503 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:57.619578 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:57.620840 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.621372 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.622937 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.623179 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.623293 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:57.625839 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:57.625848 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:57.625850 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:57.625874 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:57.627307 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:57.627824 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:57.627828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:57.627830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:57.627913 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:57.628255 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:57.628259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:57.628261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:57.628523 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:57.628630 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:57.629125 (+ 495us) server_negotiation.cc:299] Negotiation successful
0924 16:04:57.629244 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 16:04:57.630232 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:57.617528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.3:45381)
0924 16:04:57.617977 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:57.618020 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:57.619245 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:57.619760 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:57.619767 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:57.620123 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:04:57.620705 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:57.620720 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.621483 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.621485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:57.622820 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:57.622830 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.623345 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.623349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:57.623532 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:57.624136 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:04:57.624153 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:04:57.625674 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:04:57.627424 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:57.627430 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:57.627445 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:57.627715 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:57.628010 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:04:57.628013 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:04:57.628015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:04:57.628156 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:04:57.628626 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:04:57.628633 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:04:57.628881 (+ 248us) client_negotiation.cc:769] Sending connection context
0924 16:04:57.629084 (+ 203us) client_negotiation.cc:240] Negotiation successful
0924 16:04:57.629412 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":115,"threads_started":1}
I20260924 16:04:57.631337 9632 heartbeater.cc:347] Connected to a master server at 127.25.109.62:40819
I20260924 16:04:57.631603 9632 heartbeater.cc:464] Registering TS with master...
I20260924 16:04:57.632102 9632 heartbeater.cc:511] Master 127.25.109.62:40819 requested a full tablet report, sending...
I20260924 16:04:57.633191 9170 ts_manager.cc:194] Registered new tserver with Master: 3c780642dde34cc9b244869a25223c8b (127.25.109.3:35851)
I20260924 16:04:57.633764 9170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.3@KRBTEST.COM'} at 127.25.109.3:45381
I20260924 16:04:57.636303 26036 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:04:57.645433 9209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:57.638355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44584 (local address 127.25.109.62:40819)
0924 16:04:57.638537 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:57.638542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:57.638761 (+ 219us) server_negotiation.cc:407] Connection header received
0924 16:04:57.638856 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:57.638858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:57.638897 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:57.638962 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:04:57.639596 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.640084 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.640932 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.641136 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.641213 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:57.642584 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:04:57.642587 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:04:57.642589 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:04:57.642610 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:04:57.643852 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:57.644236 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:57.644239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:57.644240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:57.644310 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:04:57.644588 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:04:57.644591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:04:57.644592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:04:57.644810 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:04:57.644912 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:57.645119 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 16:04:57.645212 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:04:57.648327 9170 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:04:57.651574 9170 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:04:57.666718 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:57.660281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:35851 (local address 127.0.0.1:45726)
0924 16:04:57.661299 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:57.661315 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:57.661511 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:57.662290 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:57.662296 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:57.662331 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:57.662535 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:57.662543 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.664112 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.664115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:57.665466 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:57.665475 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.666242 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.666246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:57.666331 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:57.666353 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 16:04:57.666413 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 16:04:57.666555 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"mutex_wait_us":38,"spinlock_wait_cycles":11136,"thread_start_us":82,"threads_started":1}
I20260924 16:04:57.666723 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:57.660713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45726 (local address 127.25.109.3:35851)
0924 16:04:57.661094 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:57.661106 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:57.661385 (+ 279us) server_negotiation.cc:407] Connection header received
0924 16:04:57.661701 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:57.661708 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:57.661926 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:57.662065 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:57.662814 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.663966 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.665562 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.666109 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.666239 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:57.666391 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:57.666470 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 16:04:57.666581 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":159,"threads_started":1}
I20260924 16:04:57.666816 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:57.661446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38838 (local address 127.25.109.1:33851)
0924 16:04:57.661844 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:57.661862 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:57.662042 (+ 180us) server_negotiation.cc:407] Connection header received
0924 16:04:57.662206 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:57.662218 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:57.662360 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:57.662516 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:57.662788 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.663543 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.665086 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.665794 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.665895 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:57.666046 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:57.666527 (+ 481us) server_negotiation.cc:299] Negotiation successful
0924 16:04:57.666636 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":100,"threads_started":1}
I20260924 16:04:57.667109 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:57.660389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42292 (local address 127.25.109.2:38223)
0924 16:04:57.660922 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:57.660937 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:57.661385 (+ 448us) server_negotiation.cc:407] Connection header received
0924 16:04:57.661702 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:57.661708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:57.661926 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:57.662057 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:57.662691 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.663918 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.665502 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.666294 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.666407 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:57.666547 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:57.666871 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 16:04:57.666975 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":164,"threads_started":1}
I20260924 16:04:57.667433 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:57.661060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:33851 (local address 127.0.0.1:38838)
0924 16:04:57.661939 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:57.661950 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:57.662068 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:57.662499 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:57.662502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:57.662521 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:57.662667 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:57.662675 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.663684 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.663688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:57.664956 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:57.664966 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.666128 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.666131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:57.666253 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:57.666304 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 16:04:57.666408 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 16:04:57.667278 (+ 870us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":750,"thread_start_us":107,"threads_started":1}
I20260924 16:04:57.667637 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:57.660255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:38223 (local address 127.0.0.1:42292)
0924 16:04:57.661232 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:57.661286 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:57.661511 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:57.662290 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:57.662296 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:57.662330 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:57.662535 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:57.662543 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.664084 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.664090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:57.665368 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:57.665379 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.666619 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.666622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:57.666704 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:57.666732 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 16:04:57.667324 (+ 592us) client_negotiation.cc:240] Negotiation successful
0924 16:04:57.667439 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":109,"spinlock_wait_cycles":11776,"thread_start_us":95,"threads_started":1}
I20260924 16:04:57.670359 9295 tablet_service.cc:1467] Processing CreateTablet for tablet d58d5f5da4c74ad4a75114e06ec5a504 (DEFAULT_TABLE table=test-table [id=53bf1bb0c031454791f739acbafe828a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:04:57.670361 9430 tablet_service.cc:1467] Processing CreateTablet for tablet d58d5f5da4c74ad4a75114e06ec5a504 (DEFAULT_TABLE table=test-table [id=53bf1bb0c031454791f739acbafe828a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:04:57.671427 9430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d58d5f5da4c74ad4a75114e06ec5a504. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:57.671427 9295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d58d5f5da4c74ad4a75114e06ec5a504. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:57.672904 9565 tablet_service.cc:1467] Processing CreateTablet for tablet d58d5f5da4c74ad4a75114e06ec5a504 (DEFAULT_TABLE table=test-table [id=53bf1bb0c031454791f739acbafe828a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:04:57.673975 9565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d58d5f5da4c74ad4a75114e06ec5a504. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:04:57.678295 9649 tablet_bootstrap.cc:492] T d58d5f5da4c74ad4a75114e06ec5a504 P 3ab79c4bfa714ec89007db708ee1e66f: Bootstrap starting.
I20260924 16:04:57.680553 9649 tablet_bootstrap.cc:654] T d58d5f5da4c74ad4a75114e06ec5a504 P 3ab79c4bfa714ec89007db708ee1e66f: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:57.680783 9650 tablet_bootstrap.cc:492] T d58d5f5da4c74ad4a75114e06ec5a504 P 11f08c0f2df24d5388cedd0f60a66f92: Bootstrap starting.
I20260924 16:04:57.680882 9651 tablet_bootstrap.cc:492] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b: Bootstrap starting.
I20260924 16:04:57.682274 9649 log.cc:824] T d58d5f5da4c74ad4a75114e06ec5a504 P 3ab79c4bfa714ec89007db708ee1e66f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:57.683103 9651 tablet_bootstrap.cc:654] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:57.683525 9650 tablet_bootstrap.cc:654] T d58d5f5da4c74ad4a75114e06ec5a504 P 11f08c0f2df24d5388cedd0f60a66f92: Neither blocks nor log segments found. Creating new log.
I20260924 16:04:57.684216 9650 log.cc:824] T d58d5f5da4c74ad4a75114e06ec5a504 P 11f08c0f2df24d5388cedd0f60a66f92: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:57.684438 9651 log.cc:824] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b: Log is configured to *not* fsync() on all Append() calls
I20260924 16:04:57.684648 9649 tablet_bootstrap.cc:492] T d58d5f5da4c74ad4a75114e06ec5a504 P 3ab79c4bfa714ec89007db708ee1e66f: No bootstrap required, opened a new log
I20260924 16:04:57.684892 9649 ts_tablet_manager.cc:1397] T d58d5f5da4c74ad4a75114e06ec5a504 P 3ab79c4bfa714ec89007db708ee1e66f: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260924 16:04:57.685940 9650 tablet_bootstrap.cc:492] T d58d5f5da4c74ad4a75114e06ec5a504 P 11f08c0f2df24d5388cedd0f60a66f92: No bootstrap required, opened a new log
I20260924 16:04:57.686136 9650 ts_tablet_manager.cc:1397] T d58d5f5da4c74ad4a75114e06ec5a504 P 11f08c0f2df24d5388cedd0f60a66f92: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 16:04:57.687403 9651 tablet_bootstrap.cc:492] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b: No bootstrap required, opened a new log
I20260924 16:04:57.687600 9651 ts_tablet_manager.cc:1397] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 16:04:57.688027 9649 raft_consensus.cc:348] T d58d5f5da4c74ad4a75114e06ec5a504 P 3ab79c4bfa714ec89007db708ee1e66f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f08c0f2df24d5388cedd0f60a66f92" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38223 } } peers { permanent_uuid: "3c780642dde34cc9b244869a25223c8b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35851 } } peers { permanent_uuid: "3ab79c4bfa714ec89007db708ee1e66f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33851 } }
I20260924 16:04:57.688273 9649 raft_consensus.cc:374] T d58d5f5da4c74ad4a75114e06ec5a504 P 3ab79c4bfa714ec89007db708ee1e66f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:57.688345 9649 raft_consensus.cc:729] T d58d5f5da4c74ad4a75114e06ec5a504 P 3ab79c4bfa714ec89007db708ee1e66f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ab79c4bfa714ec89007db708ee1e66f, State: Initialized, Role: FOLLOWER
I20260924 16:04:57.688910 9649 consensus_queue.cc:260] T d58d5f5da4c74ad4a75114e06ec5a504 P 3ab79c4bfa714ec89007db708ee1e66f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f08c0f2df24d5388cedd0f60a66f92" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38223 } } peers { permanent_uuid: "3c780642dde34cc9b244869a25223c8b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35851 } } peers { permanent_uuid: "3ab79c4bfa714ec89007db708ee1e66f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33851 } }
I20260924 16:04:57.689749 9362 heartbeater.cc:503] Master 127.25.109.62:40819 was elected leader, sending a full tablet report...
I20260924 16:04:57.690101 9650 raft_consensus.cc:348] T d58d5f5da4c74ad4a75114e06ec5a504 P 11f08c0f2df24d5388cedd0f60a66f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f08c0f2df24d5388cedd0f60a66f92" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38223 } } peers { permanent_uuid: "3c780642dde34cc9b244869a25223c8b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35851 } } peers { permanent_uuid: "3ab79c4bfa714ec89007db708ee1e66f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33851 } }
I20260924 16:04:57.690371 9650 raft_consensus.cc:374] T d58d5f5da4c74ad4a75114e06ec5a504 P 11f08c0f2df24d5388cedd0f60a66f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:57.690465 9650 raft_consensus.cc:729] T d58d5f5da4c74ad4a75114e06ec5a504 P 11f08c0f2df24d5388cedd0f60a66f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11f08c0f2df24d5388cedd0f60a66f92, State: Initialized, Role: FOLLOWER
I20260924 16:04:57.690914 9650 consensus_queue.cc:260] T d58d5f5da4c74ad4a75114e06ec5a504 P 11f08c0f2df24d5388cedd0f60a66f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f08c0f2df24d5388cedd0f60a66f92" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38223 } } peers { permanent_uuid: "3c780642dde34cc9b244869a25223c8b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35851 } } peers { permanent_uuid: "3ab79c4bfa714ec89007db708ee1e66f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33851 } }
I20260924 16:04:57.691329 9649 ts_tablet_manager.cc:1428] T d58d5f5da4c74ad4a75114e06ec5a504 P 3ab79c4bfa714ec89007db708ee1e66f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 16:04:57.691563 9651 raft_consensus.cc:348] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f08c0f2df24d5388cedd0f60a66f92" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38223 } } peers { permanent_uuid: "3c780642dde34cc9b244869a25223c8b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35851 } } peers { permanent_uuid: "3ab79c4bfa714ec89007db708ee1e66f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33851 } }
I20260924 16:04:57.691864 9651 raft_consensus.cc:374] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:04:57.691949 9650 ts_tablet_manager.cc:1428] T d58d5f5da4c74ad4a75114e06ec5a504 P 11f08c0f2df24d5388cedd0f60a66f92: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 16:04:57.691941 9651 raft_consensus.cc:729] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c780642dde34cc9b244869a25223c8b, State: Initialized, Role: FOLLOWER
I20260924 16:04:57.692011 9497 heartbeater.cc:503] Master 127.25.109.62:40819 was elected leader, sending a full tablet report...
I20260924 16:04:57.692389 9651 consensus_queue.cc:260] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f08c0f2df24d5388cedd0f60a66f92" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38223 } } peers { permanent_uuid: "3c780642dde34cc9b244869a25223c8b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35851 } } peers { permanent_uuid: "3ab79c4bfa714ec89007db708ee1e66f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33851 } }
I20260924 16:04:57.693090 9632 heartbeater.cc:503] Master 127.25.109.62:40819 was elected leader, sending a full tablet report...
I20260924 16:04:57.693354 9651 ts_tablet_manager.cc:1428] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260924 16:04:57.753851 9657 raft_consensus.cc:482] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:04:57.754076 9657 raft_consensus.cc:504] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f08c0f2df24d5388cedd0f60a66f92" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38223 } } peers { permanent_uuid: "3c780642dde34cc9b244869a25223c8b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35851 } } peers { permanent_uuid: "3ab79c4bfa714ec89007db708ee1e66f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33851 } }
I20260924 16:04:57.755203 9657 leader_election.cc:288] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11f08c0f2df24d5388cedd0f60a66f92 (127.25.109.2:38223), 3ab79c4bfa714ec89007db708ee1e66f (127.25.109.1:33851)
I20260924 16:04:57.759183 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:57.755524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:38223 (local address 127.25.109.3:34081)
0924 16:04:57.755693 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:57.755710 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:57.755827 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:57.756239 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:57.756241 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:57.756259 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:57.756395 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:57.756402 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.757252 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.757258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:57.758029 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:57.758038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.758734 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.758738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:57.758855 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:57.758873 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 16:04:57.758962 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 16:04:57.759038 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 16:04:57.759183 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:57.755631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:34081 (local address 127.25.109.2:38223)
0924 16:04:57.755787 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:57.755792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:57.755811 (+ 19us) server_negotiation.cc:407] Connection header received
0924 16:04:57.755968 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:57.755972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:57.756029 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:57.756102 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:57.756569 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.757139 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.758176 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.758602 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.758734 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:57.758791 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:57.758978 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 16:04:57.759053 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 16:04:57.759449 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:57.755850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:33851 (local address 127.25.109.3:38621)
0924 16:04:57.756269 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:57.756283 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:04:57.756392 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:04:57.756716 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:04:57.756719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:04:57.756735 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:04:57.756853 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:57.756858 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.757585 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.757590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:57.758369 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:04:57.758378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.759014 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.759017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:04:57.759119 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:57.759143 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 16:04:57.759238 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 16:04:57.759307 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":132,"threads_started":1}
I20260924 16:04:57.759436 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:57.756012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:38621 (local address 127.25.109.1:33851)
0924 16:04:57.756150 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:57.756155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:57.756377 (+ 222us) server_negotiation.cc:407] Connection header received
0924 16:04:57.756536 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:57.756539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:57.756591 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:57.756670 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:04:57.756969 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.757466 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.758496 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.758887 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.759029 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:57.759081 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:57.759255 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 16:04:57.759322 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 16:04:57.760010 9450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58d5f5da4c74ad4a75114e06ec5a504" candidate_uuid: "3c780642dde34cc9b244869a25223c8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11f08c0f2df24d5388cedd0f60a66f92" is_pre_election: true
I20260924 16:04:57.760025 9316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58d5f5da4c74ad4a75114e06ec5a504" candidate_uuid: "3c780642dde34cc9b244869a25223c8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ab79c4bfa714ec89007db708ee1e66f" is_pre_election: true
I20260924 16:04:57.760325 9316 raft_consensus.cc:2454] T d58d5f5da4c74ad4a75114e06ec5a504 P 3ab79c4bfa714ec89007db708ee1e66f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c780642dde34cc9b244869a25223c8b in term 0.
I20260924 16:04:57.760329 9450 raft_consensus.cc:2454] T d58d5f5da4c74ad4a75114e06ec5a504 P 11f08c0f2df24d5388cedd0f60a66f92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c780642dde34cc9b244869a25223c8b in term 0.
I20260924 16:04:57.760865 9521 leader_election.cc:302] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ab79c4bfa714ec89007db708ee1e66f, 3c780642dde34cc9b244869a25223c8b; no voters:
I20260924 16:04:57.761165 9657 raft_consensus.cc:2781] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:04:57.761260 9657 raft_consensus.cc:482] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:04:57.761341 9657 raft_consensus.cc:3037] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:57.762430 9657 raft_consensus.cc:504] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f08c0f2df24d5388cedd0f60a66f92" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38223 } } peers { permanent_uuid: "3c780642dde34cc9b244869a25223c8b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35851 } } peers { permanent_uuid: "3ab79c4bfa714ec89007db708ee1e66f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33851 } }
I20260924 16:04:57.762778 9657 leader_election.cc:288] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b [CANDIDATE]: Term 1 election: Requested vote from peers 11f08c0f2df24d5388cedd0f60a66f92 (127.25.109.2:38223), 3ab79c4bfa714ec89007db708ee1e66f (127.25.109.1:33851)
I20260924 16:04:57.763207 9450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58d5f5da4c74ad4a75114e06ec5a504" candidate_uuid: "3c780642dde34cc9b244869a25223c8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11f08c0f2df24d5388cedd0f60a66f92"
I20260924 16:04:57.763309 9316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58d5f5da4c74ad4a75114e06ec5a504" candidate_uuid: "3c780642dde34cc9b244869a25223c8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ab79c4bfa714ec89007db708ee1e66f"
I20260924 16:04:57.763330 9450 raft_consensus.cc:3037] T d58d5f5da4c74ad4a75114e06ec5a504 P 11f08c0f2df24d5388cedd0f60a66f92 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:57.763471 9316 raft_consensus.cc:3037] T d58d5f5da4c74ad4a75114e06ec5a504 P 3ab79c4bfa714ec89007db708ee1e66f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:04:57.764565 9316 raft_consensus.cc:2454] T d58d5f5da4c74ad4a75114e06ec5a504 P 3ab79c4bfa714ec89007db708ee1e66f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c780642dde34cc9b244869a25223c8b in term 1.
I20260924 16:04:57.764578 9450 raft_consensus.cc:2454] T d58d5f5da4c74ad4a75114e06ec5a504 P 11f08c0f2df24d5388cedd0f60a66f92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c780642dde34cc9b244869a25223c8b in term 1.
I20260924 16:04:57.764894 9521 leader_election.cc:302] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11f08c0f2df24d5388cedd0f60a66f92, 3c780642dde34cc9b244869a25223c8b; no voters:
I20260924 16:04:57.765101 9657 raft_consensus.cc:2781] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:04:57.765321 9657 raft_consensus.cc:686] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b [term 1 LEADER]: Becoming Leader. State: Replica: 3c780642dde34cc9b244869a25223c8b, State: Running, Role: LEADER
I20260924 16:04:57.765661 9657 consensus_queue.cc:237] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f08c0f2df24d5388cedd0f60a66f92" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38223 } } peers { permanent_uuid: "3c780642dde34cc9b244869a25223c8b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35851 } } peers { permanent_uuid: "3ab79c4bfa714ec89007db708ee1e66f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33851 } }
I20260924 16:04:57.769187 9168 catalog_manager.cc:5495] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c780642dde34cc9b244869a25223c8b (127.25.109.3). New cstate: current_term: 1 leader_uuid: "3c780642dde34cc9b244869a25223c8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f08c0f2df24d5388cedd0f60a66f92" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c780642dde34cc9b244869a25223c8b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ab79c4bfa714ec89007db708ee1e66f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33851 } health_report { overall_health: UNKNOWN } } }
I20260924 16:04:57.788757 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:57.785346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45734 (local address 127.25.109.3:35851)
0924 16:04:57.785476 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 16:04:57.785501 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 16:04:57.785515 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:04:57.785595 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:04:57.785597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:04:57.785651 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:04:57.785738 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:04:57.785968 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.786460 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.787282 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:04:57.787503 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:04:57.787577 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:04:57.787844 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:04:57.788274 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:04:57.788365 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 16:04:57.788497 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 16:04:57.788550 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 16:04:57.794948 9450 raft_consensus.cc:1261] T d58d5f5da4c74ad4a75114e06ec5a504 P 11f08c0f2df24d5388cedd0f60a66f92 [term 1 FOLLOWER]: Refusing update from remote peer 3c780642dde34cc9b244869a25223c8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:04:57.794948 9316 raft_consensus.cc:1261] T d58d5f5da4c74ad4a75114e06ec5a504 P 3ab79c4bfa714ec89007db708ee1e66f [term 1 FOLLOWER]: Refusing update from remote peer 3c780642dde34cc9b244869a25223c8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:04:57.795763 9659 consensus_queue.cc:1035] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "11f08c0f2df24d5388cedd0f60a66f92" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:04:57.795964 9657 consensus_queue.cc:1035] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ab79c4bfa714ec89007db708ee1e66f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:04:57.805073 9663 mvcc.cc:205] Tried to move back new op lower bound from 7332929117360173056 to 7332929117251125248. Current Snapshot: MvccSnapshot[applied={T|T < 7332929117360173056}]
I20260924 16:04:57.805764 9664 mvcc.cc:205] Tried to move back new op lower bound from 7332929117360173056 to 7332929117251125248. Current Snapshot: MvccSnapshot[applied={T|T < 7332929117360173056}]
I20260924 16:04:57.812521 9170 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:04:57.812674 9170 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:04:57.815188 9170 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3: Sending DeleteTablet for 3 replicas of tablet d58d5f5da4c74ad4a75114e06ec5a504
I20260924 16:04:57.816015 9430 tablet_service.cc:1514] Processing DeleteTablet for tablet d58d5f5da4c74ad4a75114e06ec5a504 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:57 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:42292
I20260924 16:04:57.816045 9565 tablet_service.cc:1514] Processing DeleteTablet for tablet d58d5f5da4c74ad4a75114e06ec5a504 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:57 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:45726
I20260924 16:04:57.816208 9295 tablet_service.cc:1514] Processing DeleteTablet for tablet d58d5f5da4c74ad4a75114e06ec5a504 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:57 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:38838
I20260924 16:04:57.816570 9672 tablet_replica.cc:330] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b: stopping tablet replica
I20260924 16:04:57.816728 9673 tablet_replica.cc:330] T d58d5f5da4c74ad4a75114e06ec5a504 P 3ab79c4bfa714ec89007db708ee1e66f: stopping tablet replica
I20260924 16:04:57.816824 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 9232
I20260924 16:04:57.817093 9673 raft_consensus.cc:2229] T d58d5f5da4c74ad4a75114e06ec5a504 P 3ab79c4bfa714ec89007db708ee1e66f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:04:57.817607 9671 tablet_replica.cc:330] T d58d5f5da4c74ad4a75114e06ec5a504 P 11f08c0f2df24d5388cedd0f60a66f92: stopping tablet replica
I20260924 16:04:57.817888 9671 raft_consensus.cc:2229] T d58d5f5da4c74ad4a75114e06ec5a504 P 11f08c0f2df24d5388cedd0f60a66f92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:04:57.818217 9671 raft_consensus.cc:2258] T d58d5f5da4c74ad4a75114e06ec5a504 P 11f08c0f2df24d5388cedd0f60a66f92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:04:57.818696 9672 raft_consensus.cc:2229] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:04:57.819126 9672 raft_consensus.cc:2258] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:04:57.820812 9672 ts_tablet_manager.cc:1905] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:04:57.820816 9671 ts_tablet_manager.cc:1905] T d58d5f5da4c74ad4a75114e06ec5a504 P 11f08c0f2df24d5388cedd0f60a66f92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:04:57.823653 9672 ts_tablet_manager.cc:1918] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:04:57.823823 9672 log.cc:1196] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/wal/wals/d58d5f5da4c74ad4a75114e06ec5a504
I20260924 16:04:57.824232 9672 ts_tablet_manager.cc:1939] T d58d5f5da4c74ad4a75114e06ec5a504 P 3c780642dde34cc9b244869a25223c8b: Deleting consensus metadata
I20260924 16:04:57.824337 9671 ts_tablet_manager.cc:1918] T d58d5f5da4c74ad4a75114e06ec5a504 P 11f08c0f2df24d5388cedd0f60a66f92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:04:57.824558 9671 log.cc:1196] T d58d5f5da4c74ad4a75114e06ec5a504 P 11f08c0f2df24d5388cedd0f60a66f92: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/wal/wals/d58d5f5da4c74ad4a75114e06ec5a504
I20260924 16:04:57.825050 9671 ts_tablet_manager.cc:1939] T d58d5f5da4c74ad4a75114e06ec5a504 P 11f08c0f2df24d5388cedd0f60a66f92: Deleting consensus metadata
I20260924 16:04:57.825683 9155 catalog_manager.cc:4841] TS 3c780642dde34cc9b244869a25223c8b (127.25.109.3:35851): tablet d58d5f5da4c74ad4a75114e06ec5a504 (table test-table [id=53bf1bb0c031454791f739acbafe828a]) successfully deleted
W20260924 16:04:57.826098 9156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.109.1:33851 (error 108)
I20260924 16:04:57.826650 9156 catalog_manager.cc:4841] TS 11f08c0f2df24d5388cedd0f60a66f92 (127.25.109.2:38223): tablet d58d5f5da4c74ad4a75114e06ec5a504 (table test-table [id=53bf1bb0c031454791f739acbafe828a]) successfully deleted
I20260924 16:04:57.827189 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 9367
I20260924 16:04:57.827278 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:04:57.826965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:33851 (local address 127.0.0.1:38852)
0924 16:04:57.827108 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 16:04:57.827186 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.1:33851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 16:04:57.829418 9156 catalog_manager.cc:4584] TS 3ab79c4bfa714ec89007db708ee1e66f (127.25.109.1:33851): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d58d5f5da4c74ad4a75114e06ec5a504: Network error: Client connection negotiation failed: client connection to 127.25.109.1:33851: connect: Connection refused (error 111)
I20260924 16:04:57.834404 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 9502
I20260924 16:04:57.840837 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 9137
I20260924 16:05:17.848160 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:40819
--webserver_interface=127.25.109.62
--webserver_port=35681
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:40819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:17.960747 9675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:17.961086 9675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:17.961171 9675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:17.965245 9675 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 16:05:17.965325 9675 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 16:05:17.965353 9675 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 16:05:17.965386 9675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:05:17.965428 9675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:17.965474 9675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:17.965514 9675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:05:17.965550 9675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:05:17.970813 9675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:40819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:40819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=35681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:05:17.971902 9675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:17.972971 9675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:17.986526 9684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:17.986472 9681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:17.986553 9682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:17.986624 9675 server_base.cc:1034] running on GCE node
I20260924 16:05:17.987424 9675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:17.988662 9675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:17.989877 9675 hybrid_clock.cc:648] HybridClock initialized: now 1790265917989850 us; error 31 us; skew 500 ppm
Sep 24 16:05:17 dist-test-slave-n106 krb5kdc[9124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265917, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:17.992699 9675 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:05:17.993577 9675 webserver.cc:460] Webserver started at http://127.25.109.62:35681/ using document root <none> and password file <none>
I20260924 16:05:17.994192 9675 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:17.994303 9675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:17.998136 9675 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 16:05:18.000262 9691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:18.001537 9675 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:05:18.001698 9675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "2e6988e6593c40fbab8f0625315830d3"
format_stamp: "Formatted at 2026-09-24 16:04:56 on dist-test-slave-n106"
server_key: "e70a4b38b417a5725e3cbbd04d0a5d6c"
server_key_iv: "3a59fa88f803bba63b4e1a9b0f2fa4cc"
server_key_version: "encryptionkey@0"
I20260924 16:05:18.002305 9675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:18.027450 9675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:18.030498 9675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:18.030712 9675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:18.038167 9675 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:40819
I20260924 16:05:18.038465 9744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:40819 every 8 connection(s)
I20260924 16:05:18.039317 9675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:05:18.043795 9745 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:05:18.045161 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 9675
I20260924 16:05:18.045593 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:33851
--local_ip_for_outbound_sockets=127.25.109.1
--tserver_master_addrs=127.25.109.62:40819
--webserver_port=44523
--webserver_interface=127.25.109.1
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:05:18.045655 9745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3: Bootstrap starting.
I20260924 16:05:18.063696 9745 log.cc:824] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:18.072708 9745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:05:18.073113 9745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3: Bootstrap complete.
I20260924 16:05:18.077230 9745 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6988e6593c40fbab8f0625315830d3" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40819 } }
I20260924 16:05:18.077612 9745 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e6988e6593c40fbab8f0625315830d3, State: Initialized, Role: FOLLOWER
I20260924 16:05:18.078153 9745 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6988e6593c40fbab8f0625315830d3" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40819 } }
I20260924 16:05:18.078270 9745 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:05:18.078370 9745 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:05:18.078485 9745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [term 1 FOLLOWER]: Advancing to term 2
I20260924 16:05:18.080372 9745 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6988e6593c40fbab8f0625315830d3" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40819 } }
I20260924 16:05:18.080765 9745 leader_election.cc:302] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e6988e6593c40fbab8f0625315830d3; no voters:
I20260924 16:05:18.081061 9745 leader_election.cc:288] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 16:05:18.081226 9749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 16:05:18.081516 9749 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [term 2 LEADER]: Becoming Leader. State: Replica: 2e6988e6593c40fbab8f0625315830d3, State: Running, Role: LEADER
I20260924 16:05:18.081899 9749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6988e6593c40fbab8f0625315830d3" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40819 } }
I20260924 16:05:18.082286 9745 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:05:18.083382 9750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2e6988e6593c40fbab8f0625315830d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6988e6593c40fbab8f0625315830d3" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40819 } } }
I20260924 16:05:18.083518 9750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:18.083896 9751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e6988e6593c40fbab8f0625315830d3. Latest consensus state: current_term: 2 leader_uuid: "2e6988e6593c40fbab8f0625315830d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6988e6593c40fbab8f0625315830d3" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40819 } } }
I20260924 16:05:18.084067 9751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:18.084386 9758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:05:18.088240 9758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:05:18.088827 9758 catalog_manager.cc:1259] Loaded cluster ID: 5562202b9b744fc0b78d12c52e0503ce
I20260924 16:05:18.088904 9758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:05:18.090973 9758 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:05:18.091566 9758 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3: Loaded TSK: 0
I20260924 16:05:18.102968 9758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3: Generated new TSK 1
I20260924 16:05:18.103220 9758 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 16:05:18.170221 9747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:18.170542 9747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:18.170629 9747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:18.174825 9747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:18.174918 9747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:18.175025 9747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:05:18.180464 9747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:33851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=44523
--tserver_master_addrs=127.25.109.62:40819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:05:18.181661 9747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:18.182827 9747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:18.192688 9772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:18.192709 9773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:18.193118 9775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:18.193560 9747 server_base.cc:1034] running on GCE node
I20260924 16:05:18.194038 9747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:18.194833 9747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:18.196048 9747 hybrid_clock.cc:648] HybridClock initialized: now 1790265918196013 us; error 42 us; skew 500 ppm
Sep 24 16:05:18 dist-test-slave-n106 krb5kdc[9124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265918, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:18.199474 9747 init.cc:377] Logged in from keytab as kudu/127.25.109.1@KRBTEST.COM (short username kudu)
I20260924 16:05:18.200486 9747 webserver.cc:460] Webserver started at http://127.25.109.1:44523/ using document root <none> and password file <none>
I20260924 16:05:18.201064 9747 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:18.201144 9747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:18.205408 9747 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:05:18.207782 9782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:18.209192 9747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:05:18.209331 9747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "3ab79c4bfa714ec89007db708ee1e66f"
format_stamp: "Formatted at 2026-09-24 16:04:57 on dist-test-slave-n106"
server_key: "00c0523734fef87fc5f2c91428e91271"
server_key_iv: "9e2d5ce57e0f97f4a8de9c85802e0d5c"
server_key_version: "encryptionkey@0"
I20260924 16:05:18.209874 9747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:18.241206 9747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:18.244452 9747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:18.244699 9747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:18.245337 9747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:05:18.246447 9789 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 16:05:18.250607 9747 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 16:05:18.250684 9747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260924 16:05:18.250770 9747 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 16:05:18.252477 9747 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 16:05:18.252545 9747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260924 16:05:18.252858 9789 tablet_bootstrap.cc:492] T d58d5f5da4c74ad4a75114e06ec5a504 P 3ab79c4bfa714ec89007db708ee1e66f: Bootstrap starting.
I20260924 16:05:18.262796 9747 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:33851
I20260924 16:05:18.262880 9897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:33851 every 8 connection(s)
I20260924 16:05:18.263797 9747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:05:18.271243 9789 log.cc:824] T d58d5f5da4c74ad4a75114e06ec5a504 P 3ab79c4bfa714ec89007db708ee1e66f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:18.273188 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 9747
I20260924 16:05:18.273701 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:38223
--local_ip_for_outbound_sockets=127.25.109.2
--tserver_master_addrs=127.25.109.62:40819
--webserver_port=37343
--webserver_interface=127.25.109.2
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:05:18 dist-test-slave-n106 krb5kdc[9124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265918, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:05:18.279618 9789 tablet_bootstrap.cc:492] T d58d5f5da4c74ad4a75114e06ec5a504 P 3ab79c4bfa714ec89007db708ee1e66f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:05:18.280107 9789 tablet_bootstrap.cc:492] T d58d5f5da4c74ad4a75114e06ec5a504 P 3ab79c4bfa714ec89007db708ee1e66f: Bootstrap complete.
I20260924 16:05:18.281306 9900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:18.265954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:35283 (local address 127.25.109.62:40819)
0924 16:05:18.266338 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:18.266359 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:18.266697 (+ 338us) server_negotiation.cc:407] Connection header received
0924 16:05:18.267821 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:18.267832 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:18.268137 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:18.268499 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:18.269643 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.270617 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.271837 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.272318 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.272523 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:18.275105 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:18.275111 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:18.275150 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:18.275192 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:18.277238 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:18.277934 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:18.277941 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:18.277946 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:18.278122 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:18.278560 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:18.278564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:18.278566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:18.279034 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:18.279175 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:18.279976 (+ 801us) server_negotiation.cc:299] Negotiation successful
0924 16:05:18.280364 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":127,"threads_started":1}
I20260924 16:05:18.281350 9901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:18.266007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.1:35283)
0924 16:05:18.266520 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:18.266562 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:18.267391 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:18.268556 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:18.268564 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:18.268987 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:18.269449 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:18.269463 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.270767 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.270771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:18.271711 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:18.271721 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.272502 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.272507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:18.272709 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:18.273346 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:18.273371 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:18.274881 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:18.277373 (+ 2492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:18.277382 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:18.277399 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:18.277787 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:18.278245 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:18.278252 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:18.278257 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:18.278448 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:18.279173 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:18.279183 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:18.279525 (+ 342us) client_negotiation.cc:769] Sending connection context
0924 16:05:18.279789 (+ 264us) client_negotiation.cc:240] Negotiation successful
0924 16:05:18.280180 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":134,"threads_started":1}
I20260924 16:05:18.281527 9789 ts_tablet_manager.cc:1397] T d58d5f5da4c74ad4a75114e06ec5a504 P 3ab79c4bfa714ec89007db708ee1e66f: Time spent bootstrapping tablet: real 0.029s user 0.016s sys 0.012s
I20260924 16:05:18.284669 9898 heartbeater.cc:347] Connected to a master server at 127.25.109.62:40819
I20260924 16:05:18.284902 9898 heartbeater.cc:464] Registering TS with master...
I20260924 16:05:18.284524 9789 raft_consensus.cc:348] T d58d5f5da4c74ad4a75114e06ec5a504 P 3ab79c4bfa714ec89007db708ee1e66f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f08c0f2df24d5388cedd0f60a66f92" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38223 } } peers { permanent_uuid: "3c780642dde34cc9b244869a25223c8b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35851 } } peers { permanent_uuid: "3ab79c4bfa714ec89007db708ee1e66f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33851 } }
I20260924 16:05:18.285060 9789 raft_consensus.cc:729] T d58d5f5da4c74ad4a75114e06ec5a504 P 3ab79c4bfa714ec89007db708ee1e66f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ab79c4bfa714ec89007db708ee1e66f, State: Initialized, Role: FOLLOWER
I20260924 16:05:18.285475 9898 heartbeater.cc:511] Master 127.25.109.62:40819 requested a full tablet report, sending...
I20260924 16:05:18.285557 9789 consensus_queue.cc:260] T d58d5f5da4c74ad4a75114e06ec5a504 P 3ab79c4bfa714ec89007db708ee1e66f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f08c0f2df24d5388cedd0f60a66f92" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38223 } } peers { permanent_uuid: "3c780642dde34cc9b244869a25223c8b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35851 } } peers { permanent_uuid: "3ab79c4bfa714ec89007db708ee1e66f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33851 } }
I20260924 16:05:18.286620 9789 ts_tablet_manager.cc:1428] T d58d5f5da4c74ad4a75114e06ec5a504 P 3ab79c4bfa714ec89007db708ee1e66f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 16:05:18.288066 9708 ts_manager.cc:194] Registered new tserver with Master: 3ab79c4bfa714ec89007db708ee1e66f (127.25.109.1:33851)
I20260924 16:05:18.291605 9708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.1@KRBTEST.COM'} at 127.25.109.1:35283
I20260924 16:05:18.294350 9898 heartbeater.cc:503] Master 127.25.109.62:40819 was elected leader, sending a full tablet report...
Sep 24 16:05:18 dist-test-slave-n106 krb5kdc[9124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265917, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for kudu/127.25.109.1@KRBTEST.COM
I20260924 16:05:18.302220 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:18.290880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:33851 (local address 127.0.0.1:41334)
0924 16:05:18.291323 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:18.291370 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:18.291499 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:18.292145 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:18.292151 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:18.292638 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:18.292770 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:18.292778 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.294080 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.294083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:18.295086 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:18.295098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.296178 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.296181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:18.296295 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:18.296850 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:18.296866 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:18.298391 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:18.300348 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:18.300357 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:18.300364 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:18.300649 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:18.300967 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:18.300970 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:18.300973 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:18.301073 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:18.301560 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:18.301566 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:18.301693 (+ 127us) client_negotiation.cc:769] Sending connection context
0924 16:05:18.301830 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 16:05:18.302035 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":130,"threads_started":1}
I20260924 16:05:18.302253 9906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:18.290978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41334 (local address 127.25.109.1:33851)
0924 16:05:18.291653 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:18.291665 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:18.291690 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:05:18.291764 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:18.291769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:18.291980 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:18.292124 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:18.292904 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.293947 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.295216 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.295750 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.296005 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:18.298541 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:18.298557 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:18.298568 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:18.298590 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:18.300210 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:18.300783 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:18.300788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:18.300790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:18.300869 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:18.301174 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:18.301177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:18.301179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:18.301426 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:18.301573 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:18.301930 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 16:05:18.302105 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":124,"threads_started":1}
I20260924 16:05:18.303301 9831 tablet_service.cc:1514] Processing DeleteTablet for tablet d58d5f5da4c74ad4a75114e06ec5a504 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:04:57 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:41334
I20260924 16:05:18.304330 9907 tablet_replica.cc:330] stopping tablet replica
I20260924 16:05:18.304666 9907 raft_consensus.cc:2229] T d58d5f5da4c74ad4a75114e06ec5a504 P 3ab79c4bfa714ec89007db708ee1e66f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:05:18.304931 9907 raft_consensus.cc:2258] T d58d5f5da4c74ad4a75114e06ec5a504 P 3ab79c4bfa714ec89007db708ee1e66f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:05:18.305994 9907 ts_tablet_manager.cc:1905] T d58d5f5da4c74ad4a75114e06ec5a504 P 3ab79c4bfa714ec89007db708ee1e66f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:05:18.309607 9907 ts_tablet_manager.cc:1918] T d58d5f5da4c74ad4a75114e06ec5a504 P 3ab79c4bfa714ec89007db708ee1e66f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:05:18.309717 9907 log.cc:1196] T d58d5f5da4c74ad4a75114e06ec5a504 P 3ab79c4bfa714ec89007db708ee1e66f: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/wal/wals/d58d5f5da4c74ad4a75114e06ec5a504
I20260924 16:05:18.309988 9907 ts_tablet_manager.cc:1939] T d58d5f5da4c74ad4a75114e06ec5a504 P 3ab79c4bfa714ec89007db708ee1e66f: Deleting consensus metadata
I20260924 16:05:18.311641 9695 catalog_manager.cc:4841] TS 3ab79c4bfa714ec89007db708ee1e66f (127.25.109.1:33851): tablet d58d5f5da4c74ad4a75114e06ec5a504 (table test-table [id=53bf1bb0c031454791f739acbafe828a]) successfully deleted
W20260924 16:05:18.393896 9903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:18.394160 9903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:18.394243 9903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:18.398241 9903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:18.398296 9903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:18.398406 9903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:05:18.403764 9903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:38223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=37343
--tserver_master_addrs=127.25.109.62:40819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:05:18.404923 9903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:18.406064 9903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:18.415768 9915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:18.415783 9913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:18.415776 9912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:18.416121 9903 server_base.cc:1034] running on GCE node
I20260924 16:05:18.416568 9903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:18.417125 9903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:18.418277 9903 hybrid_clock.cc:648] HybridClock initialized: now 1790265918418245 us; error 43 us; skew 500 ppm
Sep 24 16:05:18 dist-test-slave-n106 krb5kdc[9124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265918, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:18.421147 9903 init.cc:377] Logged in from keytab as kudu/127.25.109.2@KRBTEST.COM (short username kudu)
I20260924 16:05:18.422403 9903 webserver.cc:460] Webserver started at http://127.25.109.2:37343/ using document root <none> and password file <none>
I20260924 16:05:18.423235 9903 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:18.423378 9903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:18.428700 9903 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:05:18.431143 9922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:18.432462 9903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:05:18.432614 9903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "11f08c0f2df24d5388cedd0f60a66f92"
format_stamp: "Formatted at 2026-09-24 16:04:57 on dist-test-slave-n106"
server_key: "6f297cfe1d5d9fe15aafc2974ebc4387"
server_key_iv: "59c983a953a0137da7154db4c6f280fe"
server_key_version: "encryptionkey@0"
I20260924 16:05:18.433171 9903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:18.447037 9903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:18.450002 9903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:18.450219 9903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:18.450835 9903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:05:18.451694 9903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:18.451762 9903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:18.451833 9903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:18.451876 9903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:18.460666 9903 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:38223
I20260924 16:05:18.460690 10036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:38223 every 8 connection(s)
I20260924 16:05:18.461728 9903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:05:18.462193 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 9903
I20260924 16:05:18.462728 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:35851
--local_ip_for_outbound_sockets=127.25.109.3
--tserver_master_addrs=127.25.109.62:40819
--webserver_port=34043
--webserver_interface=127.25.109.3
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:05:18 dist-test-slave-n106 krb5kdc[9124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265918, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:05:18.478063 9900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:18.463814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:33765 (local address 127.25.109.62:40819)
0924 16:05:18.463957 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:18.463961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:18.465298 (+ 1337us) server_negotiation.cc:407] Connection header received
0924 16:05:18.466645 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:18.466648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:18.466700 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:18.466767 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:18.468526 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.469056 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.470992 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.471235 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.471370 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:18.474346 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:18.474350 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:18.474352 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:18.474378 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:18.475885 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:18.476442 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:18.476446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:18.476447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:18.476536 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:18.476910 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:18.476914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:18.476915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:18.477111 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:18.477243 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:18.477740 (+ 497us) server_negotiation.cc:299] Negotiation successful
0924 16:05:18.477883 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 16:05:18.478868 10040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:18.464344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.2:33765)
0924 16:05:18.465118 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:18.465171 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:18.466392 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:18.466962 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:18.466974 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:18.467528 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:18.468313 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:18.468338 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.469204 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.469209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:18.470833 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:18.470848 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.471439 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.471445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:18.471704 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:18.472379 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:18.472431 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:18.474160 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:18.476022 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:18.476028 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:18.476043 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:18.476310 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:18.476651 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:18.476654 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:18.476656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:18.476802 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:18.477240 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:18.477247 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:18.477497 (+ 250us) client_negotiation.cc:769] Sending connection context
0924 16:05:18.477682 (+ 185us) client_negotiation.cc:240] Negotiation successful
0924 16:05:18.478010 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":142,"threads_started":1}
I20260924 16:05:18.480223 10037 heartbeater.cc:347] Connected to a master server at 127.25.109.62:40819
I20260924 16:05:18.480523 10037 heartbeater.cc:464] Registering TS with master...
I20260924 16:05:18.481032 10037 heartbeater.cc:511] Master 127.25.109.62:40819 requested a full tablet report, sending...
I20260924 16:05:18.482172 9708 ts_manager.cc:194] Registered new tserver with Master: 11f08c0f2df24d5388cedd0f60a66f92 (127.25.109.2:38223)
I20260924 16:05:18.482980 9708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.2@KRBTEST.COM'} at 127.25.109.2:33765
W20260924 16:05:18.578250 10039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:18.578493 10039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:18.578534 10039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:18.582641 10039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:18.582700 10039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:18.582758 10039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:05:18.588212 10039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:35851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=34043
--tserver_master_addrs=127.25.109.62:40819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:05:18.589370 10039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:18.590322 10039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:05:18.599890 10039 server_base.cc:1034] running on GCE node
W20260924 16:05:18.599852 10046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:18.599840 10045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:18.599823 10048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:18.600520 10039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:18.601163 10039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:18.602341 10039 hybrid_clock.cc:648] HybridClock initialized: now 1790265918602328 us; error 25 us; skew 500 ppm
Sep 24 16:05:18 dist-test-slave-n106 krb5kdc[9124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265918, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:18.604911 10039 init.cc:377] Logged in from keytab as kudu/127.25.109.3@KRBTEST.COM (short username kudu)
I20260924 16:05:18.605829 10039 webserver.cc:460] Webserver started at http://127.25.109.3:34043/ using document root <none> and password file <none>
I20260924 16:05:18.606441 10039 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:18.606539 10039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:18.610380 10039 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:05:18.612512 10055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:18.613782 10039 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:05:18.613946 10039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "3c780642dde34cc9b244869a25223c8b"
format_stamp: "Formatted at 2026-09-24 16:04:57 on dist-test-slave-n106"
server_key: "7d8b653ade785dfeeaac4f05d9bcbac3"
server_key_iv: "39f7befb2b91d0ce6f5bcab435f4e500"
server_key_version: "encryptionkey@0"
I20260924 16:05:18.614562 10039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:18.623610 10039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:18.626345 10039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:18.626530 10039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:18.627099 10039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:05:18.627892 10039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:18.627945 10039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:18.628011 10039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:18.628048 10039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:18.636819 10039 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:35851
I20260924 16:05:18.636833 10170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:35851 every 8 connection(s)
I20260924 16:05:18.637982 10039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:05:18.639990 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 10039
Sep 24 16:05:18 dist-test-slave-n106 krb5kdc[9124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265918, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:05:18.650110 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:18.641314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34920 (local address 127.25.109.62:40819)
0924 16:05:18.641552 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:18.641556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:18.641567 (+ 11us) server_negotiation.cc:407] Connection header received
0924 16:05:18.641606 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:18.641608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:18.641647 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:18.641720 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:18.642807 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.643377 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.644364 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.644676 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.644755 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:18.646296 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:18.646299 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:18.646301 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:18.646319 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:18.648234 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:18.648723 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:18.648727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:18.648729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:18.648818 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:18.649099 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:18.649101 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:18.649102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:18.649357 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:18.649550 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:18.649797 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 16:05:18.649949 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":103,"threads_started":1}
I20260924 16:05:18.652861 9900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:18.639764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:53663 (local address 127.25.109.62:40819)
0924 16:05:18.639909 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:18.639913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:18.641255 (+ 1342us) server_negotiation.cc:407] Connection header received
0924 16:05:18.642349 (+ 1094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:18.642351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:18.642392 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:18.642498 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:18.643772 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.644257 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.645831 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.646038 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.646114 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:18.648947 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:18.648954 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:18.648960 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:18.648988 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:18.650726 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:18.651270 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:18.651273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:18.651275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:18.651357 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:18.651727 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:18.651730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:18.651732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:18.651966 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:18.652065 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:18.652608 (+ 543us) server_negotiation.cc:299] Negotiation successful
0924 16:05:18.652717 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 16:05:18.653688 10173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:18.640274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.3:53663)
0924 16:05:18.641116 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:18.641156 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:18.642099 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:18.642732 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:18.642739 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:18.643063 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:18.643606 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:18.643635 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.644451 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.644454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:18.645699 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:18.645711 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.646187 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.646193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:18.646382 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:18.647070 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:18.647085 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:18.648769 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:18.650841 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:18.650847 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:18.650862 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:18.651157 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:18.651475 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:18.651478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:18.651480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:18.651621 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:18.652064 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:18.652071 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:18.652310 (+ 239us) client_negotiation.cc:769] Sending connection context
0924 16:05:18.652551 (+ 241us) client_negotiation.cc:240] Negotiation successful
0924 16:05:18.652878 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":155,"threads_started":1}
I20260924 16:05:18.654740 10171 heartbeater.cc:347] Connected to a master server at 127.25.109.62:40819
I20260924 16:05:18.654982 10171 heartbeater.cc:464] Registering TS with master...
I20260924 16:05:18.655458 10171 heartbeater.cc:511] Master 127.25.109.62:40819 requested a full tablet report, sending...
I20260924 16:05:18.656481 9708 ts_manager.cc:194] Registered new tserver with Master: 3c780642dde34cc9b244869a25223c8b (127.25.109.3:35851)
I20260924 16:05:18.657024 9708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.3@KRBTEST.COM'} at 127.25.109.3:53663
I20260924 16:05:18.663601 26036 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:05:18.667366 9900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:18.664222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34924 (local address 127.25.109.62:40819)
0924 16:05:18.664356 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:18.664360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:18.664616 (+ 256us) server_negotiation.cc:407] Connection header received
0924 16:05:18.664736 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:18.664738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:18.664829 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:18.664915 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:18.665124 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.665599 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.666387 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.666587 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.666662 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:18.666914 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:18.667039 (+ 125us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 16:05:18.667237 (+ 198us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34924: token expired
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 16:05:18.667435 9900 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34924: token expired
W20260924 16:05:18.667807 9638 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.25.109.62:40819)
I20260924 16:05:18.674445 9900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:18.668161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34936 (local address 127.25.109.62:40819)
0924 16:05:18.668296 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:18.668299 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:18.668312 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:05:18.668342 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:18.668344 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:18.668376 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:18.668540 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:18.669100 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.669553 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.670294 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.670484 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.670553 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:18.671833 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:18.671836 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:18.671838 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:18.671855 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:18.673166 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:18.673527 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:18.673530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:18.673531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:18.673598 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:18.673831 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:18.673833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:18.673834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:18.673978 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:18.674072 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:18.674261 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 16:05:18.674342 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 16:05:18.676822 9708 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:05:18.680047 9708 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:05:18.688333 9831 tablet_service.cc:1467] Processing CreateTablet for tablet 7f06daada7c744d78b792be13e8d2635 (DEFAULT_TABLE table=test-table [id=78a12a4b918d4aa88eae511e15482baf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:05:18.689116 9831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f06daada7c744d78b792be13e8d2635. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:18.692088 9789 tablet_bootstrap.cc:492] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f: Bootstrap starting.
I20260924 16:05:18.692767 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:18.686993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:35851 (local address 127.0.0.1:45718)
0924 16:05:18.687119 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:18.687136 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:18.687223 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:18.688249 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:18.688252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:18.688271 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:18.688381 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:18.688388 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.689794 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.689798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:18.690982 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:18.690996 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.692303 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.692306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:18.692422 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:18.692469 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 16:05:18.692551 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 16:05:18.692635 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 16:05:18.693063 9789 tablet_bootstrap.cc:654] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:18.693140 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:18.687332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45718 (local address 127.25.109.3:35851)
0924 16:05:18.687753 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:18.687770 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:18.687796 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:05:18.687880 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:18.687885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:18.688094 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:18.688275 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:18.688544 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.689672 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.691134 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.691984 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.692086 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:18.692246 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:18.692611 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 16:05:18.692750 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":186,"threads_started":1}
I20260924 16:05:18.694590 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:18.687428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:38223 (local address 127.0.0.1:50708)
0924 16:05:18.688543 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:18.688556 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:18.688651 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:18.689604 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:18.689607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:18.689632 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:18.689727 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:18.689737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.690876 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.690882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:18.692130 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:18.692140 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.694196 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.694200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:18.694301 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:18.694324 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 16:05:18.694402 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 16:05:18.694465 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"thread_start_us":126,"threads_started":1}
I20260924 16:05:18.694670 9789 tablet_bootstrap.cc:492] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f: No bootstrap required, opened a new log
I20260924 16:05:18.694715 10182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:18.687655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50708 (local address 127.25.109.2:38223)
0924 16:05:18.687943 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:18.687957 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:18.688816 (+ 859us) server_negotiation.cc:407] Connection header received
0924 16:05:18.689124 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:18.689131 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:18.689310 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:18.689428 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:18.689930 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.690759 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.692262 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.693161 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.693986 (+ 825us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:18.694138 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:18.694464 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 16:05:18.694567 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":107,"threads_started":1}
I20260924 16:05:18.694752 9789 ts_tablet_manager.cc:1397] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 16:05:18.695232 9789 raft_consensus.cc:348] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c780642dde34cc9b244869a25223c8b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35851 } } peers { permanent_uuid: "11f08c0f2df24d5388cedd0f60a66f92" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38223 } } peers { permanent_uuid: "3ab79c4bfa714ec89007db708ee1e66f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33851 } }
I20260924 16:05:18.695343 9789 raft_consensus.cc:374] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:18.695384 9789 raft_consensus.cc:729] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ab79c4bfa714ec89007db708ee1e66f, State: Initialized, Role: FOLLOWER
I20260924 16:05:18.695518 9789 consensus_queue.cc:260] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c780642dde34cc9b244869a25223c8b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35851 } } peers { permanent_uuid: "11f08c0f2df24d5388cedd0f60a66f92" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38223 } } peers { permanent_uuid: "3ab79c4bfa714ec89007db708ee1e66f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33851 } }
I20260924 16:05:18.696086 10103 tablet_service.cc:1467] Processing CreateTablet for tablet 7f06daada7c744d78b792be13e8d2635 (DEFAULT_TABLE table=test-table [id=78a12a4b918d4aa88eae511e15482baf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:05:18.696661 9789 ts_tablet_manager.cc:1428] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 16:05:18.697256 10103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f06daada7c744d78b792be13e8d2635. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:18.698455 9970 tablet_service.cc:1467] Processing CreateTablet for tablet 7f06daada7c744d78b792be13e8d2635 (DEFAULT_TABLE table=test-table [id=78a12a4b918d4aa88eae511e15482baf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:05:18.699465 9970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f06daada7c744d78b792be13e8d2635. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:18.705394 10184 tablet_bootstrap.cc:492] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b: Bootstrap starting.
I20260924 16:05:18.706916 10185 tablet_bootstrap.cc:492] T 7f06daada7c744d78b792be13e8d2635 P 11f08c0f2df24d5388cedd0f60a66f92: Bootstrap starting.
I20260924 16:05:18.708101 10184 tablet_bootstrap.cc:654] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:18.708964 10184 log.cc:824] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:18.709335 10185 tablet_bootstrap.cc:654] T 7f06daada7c744d78b792be13e8d2635 P 11f08c0f2df24d5388cedd0f60a66f92: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:18.710196 10185 log.cc:824] T 7f06daada7c744d78b792be13e8d2635 P 11f08c0f2df24d5388cedd0f60a66f92: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:18.711050 10184 tablet_bootstrap.cc:492] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b: No bootstrap required, opened a new log
I20260924 16:05:18.711223 10184 ts_tablet_manager.cc:1397] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 16:05:18.711961 10185 tablet_bootstrap.cc:492] T 7f06daada7c744d78b792be13e8d2635 P 11f08c0f2df24d5388cedd0f60a66f92: No bootstrap required, opened a new log
I20260924 16:05:18.712159 10185 ts_tablet_manager.cc:1397] T 7f06daada7c744d78b792be13e8d2635 P 11f08c0f2df24d5388cedd0f60a66f92: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 16:05:18.714630 10184 raft_consensus.cc:348] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c780642dde34cc9b244869a25223c8b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35851 } } peers { permanent_uuid: "11f08c0f2df24d5388cedd0f60a66f92" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38223 } } peers { permanent_uuid: "3ab79c4bfa714ec89007db708ee1e66f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33851 } }
I20260924 16:05:18.714866 10184 raft_consensus.cc:374] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:18.714949 10184 raft_consensus.cc:729] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c780642dde34cc9b244869a25223c8b, State: Initialized, Role: FOLLOWER
I20260924 16:05:18.715444 10184 consensus_queue.cc:260] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c780642dde34cc9b244869a25223c8b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35851 } } peers { permanent_uuid: "11f08c0f2df24d5388cedd0f60a66f92" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38223 } } peers { permanent_uuid: "3ab79c4bfa714ec89007db708ee1e66f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33851 } }
I20260924 16:05:18.715392 10185 raft_consensus.cc:348] T 7f06daada7c744d78b792be13e8d2635 P 11f08c0f2df24d5388cedd0f60a66f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c780642dde34cc9b244869a25223c8b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35851 } } peers { permanent_uuid: "11f08c0f2df24d5388cedd0f60a66f92" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38223 } } peers { permanent_uuid: "3ab79c4bfa714ec89007db708ee1e66f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33851 } }
I20260924 16:05:18.715646 10185 raft_consensus.cc:374] T 7f06daada7c744d78b792be13e8d2635 P 11f08c0f2df24d5388cedd0f60a66f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:18.715709 10185 raft_consensus.cc:729] T 7f06daada7c744d78b792be13e8d2635 P 11f08c0f2df24d5388cedd0f60a66f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11f08c0f2df24d5388cedd0f60a66f92, State: Initialized, Role: FOLLOWER
I20260924 16:05:18.716144 10171 heartbeater.cc:503] Master 127.25.109.62:40819 was elected leader, sending a full tablet report...
I20260924 16:05:18.716542 10184 ts_tablet_manager.cc:1428] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260924 16:05:18.717248 10185 consensus_queue.cc:260] T 7f06daada7c744d78b792be13e8d2635 P 11f08c0f2df24d5388cedd0f60a66f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c780642dde34cc9b244869a25223c8b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35851 } } peers { permanent_uuid: "11f08c0f2df24d5388cedd0f60a66f92" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38223 } } peers { permanent_uuid: "3ab79c4bfa714ec89007db708ee1e66f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33851 } }
I20260924 16:05:18.717942 10037 heartbeater.cc:503] Master 127.25.109.62:40819 was elected leader, sending a full tablet report...
I20260924 16:05:18.718281 10185 ts_tablet_manager.cc:1428] T 7f06daada7c744d78b792be13e8d2635 P 11f08c0f2df24d5388cedd0f60a66f92: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260924 16:05:18.765108 9899 tablet.cc:2369] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:05:18.779294 10188 raft_consensus.cc:482] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:05:18.779600 10188 raft_consensus.cc:504] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c780642dde34cc9b244869a25223c8b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35851 } } peers { permanent_uuid: "11f08c0f2df24d5388cedd0f60a66f92" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38223 } } peers { permanent_uuid: "3ab79c4bfa714ec89007db708ee1e66f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33851 } }
I20260924 16:05:18.781091 10188 leader_election.cc:288] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11f08c0f2df24d5388cedd0f60a66f92 (127.25.109.2:38223), 3ab79c4bfa714ec89007db708ee1e66f (127.25.109.1:33851)
I20260924 16:05:18.786468 10173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:18.781340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:38223 (local address 127.25.109.3:36233)
0924 16:05:18.781473 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:18.781492 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:18.781603 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:18.781951 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:18.781954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:18.781975 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:18.782097 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:18.782104 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.783229 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.783232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:18.784466 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:18.784477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.786049 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.786053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:18.786159 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:18.786181 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 16:05:18.786247 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 16:05:18.786322 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":21}
I20260924 16:05:18.786639 9906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:18.782046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:43303 (local address 127.25.109.1:33851)
0924 16:05:18.782202 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:18.782206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:18.782220 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:05:18.782269 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:18.782272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:18.782350 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:18.782445 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:18.782720 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.783499 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.785189 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.785842 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.785958 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:18.786124 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:18.786433 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 16:05:18.786518 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 16:05:18.786844 10182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:18.781470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:36233 (local address 127.25.109.2:38223)
0924 16:05:18.781611 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:18.781616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:18.781634 (+ 18us) server_negotiation.cc:407] Connection header received
0924 16:05:18.781694 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:18.781697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:18.781744 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:18.781816 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:18.782246 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.783049 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.785189 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.785816 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.785933 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:18.785987 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:18.786651 (+ 664us) server_negotiation.cc:299] Negotiation successful
0924 16:05:18.786734 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 16:05:18.787432 10190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:18.781641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:33851 (local address 127.25.109.3:43303)
0924 16:05:18.782011 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:18.782025 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:18.782124 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:18.782469 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:18.782472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:18.782501 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:18.782622 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:18.782628 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.783601 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.783605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:18.785036 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:18.785043 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.786192 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.786195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:18.786300 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:18.786322 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 16:05:18.787216 (+ 894us) client_negotiation.cc:240] Negotiation successful
0924 16:05:18.787296 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":1,"thread_start_us":127,"threads_started":1}
I20260924 16:05:18.787463 9991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f06daada7c744d78b792be13e8d2635" candidate_uuid: "3c780642dde34cc9b244869a25223c8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11f08c0f2df24d5388cedd0f60a66f92" is_pre_election: true
I20260924 16:05:18.787807 9991 raft_consensus.cc:2454] T 7f06daada7c744d78b792be13e8d2635 P 11f08c0f2df24d5388cedd0f60a66f92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c780642dde34cc9b244869a25223c8b in term 0.
I20260924 16:05:18.788218 9852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f06daada7c744d78b792be13e8d2635" candidate_uuid: "3c780642dde34cc9b244869a25223c8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ab79c4bfa714ec89007db708ee1e66f" is_pre_election: true
I20260924 16:05:18.788537 9852 raft_consensus.cc:2454] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c780642dde34cc9b244869a25223c8b in term 0.
I20260924 16:05:18.789057 10059 leader_election.cc:302] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11f08c0f2df24d5388cedd0f60a66f92, 3c780642dde34cc9b244869a25223c8b; no voters:
I20260924 16:05:18.789546 10188 raft_consensus.cc:2781] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:05:18.789659 10188 raft_consensus.cc:482] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:05:18.789709 10188 raft_consensus.cc:3037] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:18.790916 10188 raft_consensus.cc:504] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c780642dde34cc9b244869a25223c8b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35851 } } peers { permanent_uuid: "11f08c0f2df24d5388cedd0f60a66f92" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38223 } } peers { permanent_uuid: "3ab79c4bfa714ec89007db708ee1e66f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33851 } }
I20260924 16:05:18.791301 10188 leader_election.cc:288] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b [CANDIDATE]: Term 1 election: Requested vote from peers 11f08c0f2df24d5388cedd0f60a66f92 (127.25.109.2:38223), 3ab79c4bfa714ec89007db708ee1e66f (127.25.109.1:33851)
I20260924 16:05:18.791723 9991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f06daada7c744d78b792be13e8d2635" candidate_uuid: "3c780642dde34cc9b244869a25223c8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11f08c0f2df24d5388cedd0f60a66f92"
I20260924 16:05:18.791782 9852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f06daada7c744d78b792be13e8d2635" candidate_uuid: "3c780642dde34cc9b244869a25223c8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ab79c4bfa714ec89007db708ee1e66f"
I20260924 16:05:18.791877 9991 raft_consensus.cc:3037] T 7f06daada7c744d78b792be13e8d2635 P 11f08c0f2df24d5388cedd0f60a66f92 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:18.791922 9852 raft_consensus.cc:3037] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:18.793157 9991 raft_consensus.cc:2454] T 7f06daada7c744d78b792be13e8d2635 P 11f08c0f2df24d5388cedd0f60a66f92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c780642dde34cc9b244869a25223c8b in term 1.
I20260924 16:05:18.793226 9852 raft_consensus.cc:2454] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c780642dde34cc9b244869a25223c8b in term 1.
I20260924 16:05:18.793543 10059 leader_election.cc:302] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ab79c4bfa714ec89007db708ee1e66f, 3c780642dde34cc9b244869a25223c8b; no voters:
I20260924 16:05:18.793727 10188 raft_consensus.cc:2781] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:05:18.793958 10188 raft_consensus.cc:686] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b [term 1 LEADER]: Becoming Leader. State: Replica: 3c780642dde34cc9b244869a25223c8b, State: Running, Role: LEADER
I20260924 16:05:18.794337 10188 consensus_queue.cc:237] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c780642dde34cc9b244869a25223c8b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35851 } } peers { permanent_uuid: "11f08c0f2df24d5388cedd0f60a66f92" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38223 } } peers { permanent_uuid: "3ab79c4bfa714ec89007db708ee1e66f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33851 } }
I20260924 16:05:18.797799 9708 catalog_manager.cc:5495] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c780642dde34cc9b244869a25223c8b (127.25.109.3). New cstate: current_term: 1 leader_uuid: "3c780642dde34cc9b244869a25223c8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c780642dde34cc9b244869a25223c8b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11f08c0f2df24d5388cedd0f60a66f92" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ab79c4bfa714ec89007db708ee1e66f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33851 } health_report { overall_health: UNKNOWN } } }
I20260924 16:05:18.819888 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:18.816145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45732 (local address 127.25.109.3:35851)
0924 16:05:18.816282 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:18.816286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:18.816301 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:05:18.816499 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:18.816502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:18.816562 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:18.816666 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:18.816882 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.817476 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.818445 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:18.818675 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:18.818759 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:18.819062 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:18.819523 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:18.819611 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:18.819699 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 16:05:18.819752 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 16:05:18.826094 9852 raft_consensus.cc:1261] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [term 1 FOLLOWER]: Refusing update from remote peer 3c780642dde34cc9b244869a25223c8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:05:18.826098 9991 raft_consensus.cc:1261] T 7f06daada7c744d78b792be13e8d2635 P 11f08c0f2df24d5388cedd0f60a66f92 [term 1 FOLLOWER]: Refusing update from remote peer 3c780642dde34cc9b244869a25223c8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:05:18.826778 10188 consensus_queue.cc:1035] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ab79c4bfa714ec89007db708ee1e66f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:05:18.826972 10191 consensus_queue.cc:1035] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "11f08c0f2df24d5388cedd0f60a66f92" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:05:18.832669 10193 mvcc.cc:205] Tried to move back new op lower bound from 7332929203504177152 to 7332929203384508416. Current Snapshot: MvccSnapshot[applied={T|T < 7332929203504177152}]
I20260924 16:05:18.833896 10194 mvcc.cc:205] Tried to move back new op lower bound from 7332929203504177152 to 7332929203384508416. Current Snapshot: MvccSnapshot[applied={T|T < 7332929203504177152}]
I20260924 16:05:18.843096 9707 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:05:18.843299 9707 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:05:18.845917 9707 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3: Sending DeleteTablet for 3 replicas of tablet 7f06daada7c744d78b792be13e8d2635
I20260924 16:05:18.846807 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 9747
I20260924 16:05:18.846853 9970 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f06daada7c744d78b792be13e8d2635 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:05:18 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:50708
I20260924 16:05:18.847466 10103 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f06daada7c744d78b792be13e8d2635 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:05:18 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:45718
I20260924 16:05:18.848639 10202 tablet_replica.cc:330] T 7f06daada7c744d78b792be13e8d2635 P 11f08c0f2df24d5388cedd0f60a66f92: stopping tablet replica
I20260924 16:05:18.849030 10202 raft_consensus.cc:2229] T 7f06daada7c744d78b792be13e8d2635 P 11f08c0f2df24d5388cedd0f60a66f92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:05:18.849371 10202 raft_consensus.cc:2258] T 7f06daada7c744d78b792be13e8d2635 P 11f08c0f2df24d5388cedd0f60a66f92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:05:18.849777 10203 tablet_replica.cc:330] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b: stopping tablet replica
I20260924 16:05:18.850126 10203 raft_consensus.cc:2229] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:05:18.850533 10203 raft_consensus.cc:2258] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:05:18.851183 10202 ts_tablet_manager.cc:1905] T 7f06daada7c744d78b792be13e8d2635 P 11f08c0f2df24d5388cedd0f60a66f92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:05:18.851989 10203 ts_tablet_manager.cc:1905] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:05:18.854682 10203 ts_tablet_manager.cc:1918] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:05:18.854803 10202 ts_tablet_manager.cc:1918] T 7f06daada7c744d78b792be13e8d2635 P 11f08c0f2df24d5388cedd0f60a66f92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:05:18.854847 10203 log.cc:1196] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/wal/wals/7f06daada7c744d78b792be13e8d2635
I20260924 16:05:18.855018 10202 log.cc:1196] T 7f06daada7c744d78b792be13e8d2635 P 11f08c0f2df24d5388cedd0f60a66f92: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/wal/wals/7f06daada7c744d78b792be13e8d2635
W20260924 16:05:18.855023 9695 connection.cc:535] client connection to 127.25.109.1:33851 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 16:05:18.855180 9695 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 16:05:18.855490 10202 ts_tablet_manager.cc:1939] T 7f06daada7c744d78b792be13e8d2635 P 11f08c0f2df24d5388cedd0f60a66f92: Deleting consensus metadata
I20260924 16:05:18.855700 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 9903
I20260924 16:05:18.855861 10203 ts_tablet_manager.cc:1939] T 7f06daada7c744d78b792be13e8d2635 P 3c780642dde34cc9b244869a25223c8b: Deleting consensus metadata
I20260924 16:05:18.857199 9694 catalog_manager.cc:4841] TS 3c780642dde34cc9b244869a25223c8b (127.25.109.3:35851): tablet 7f06daada7c744d78b792be13e8d2635 (table test-table [id=78a12a4b918d4aa88eae511e15482baf]) successfully deleted
I20260924 16:05:18.861327 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:18.861011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:33851 (local address 127.0.0.1:41344)
0924 16:05:18.861156 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:18.861238 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.1:33851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 16:05:18.861580 9695 catalog_manager.cc:4584] TS 3ab79c4bfa714ec89007db708ee1e66f (127.25.109.1:33851): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7f06daada7c744d78b792be13e8d2635: Network error: Client connection negotiation failed: client connection to 127.25.109.1:33851: connect: Connection refused (error 111)
I20260924 16:05:18.862916 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 10039
I20260924 16:05:18.864619 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:18.863048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:38223 (local address 127.0.0.1:50710)
0924 16:05:18.864464 (+ 1416us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:18.864530 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.2:38223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1301}
I20260924 16:05:18.868357 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 9675
I20260924 16:05:38.875206 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:40819
--webserver_interface=127.25.109.62
--webserver_port=35681
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:40819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:38.987105 10205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:38.987358 10205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:38.987401 10205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:38.991372 10205 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 16:05:38.991437 10205 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 16:05:38.991461 10205 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 16:05:38.991475 10205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:05:38.991492 10205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:38.991511 10205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:38.991526 10205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:05:38.991541 10205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:05:38.996709 10205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:40819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:40819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=35681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:05:38.997701 10205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:38.998677 10205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:39.007920 10211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:39.007962 10205 server_base.cc:1034] running on GCE node
W20260924 16:05:39.007910 10210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:39.007901 10213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:39.008716 10205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:39.009936 10205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:39.011159 10205 hybrid_clock.cc:648] HybridClock initialized: now 1790265939011127 us; error 63 us; skew 500 ppm
Sep 24 16:05:39 dist-test-slave-n106 krb5kdc[9124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265939, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:39.014070 10205 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:05:39.014917 10205 webserver.cc:460] Webserver started at http://127.25.109.62:35681/ using document root <none> and password file <none>
I20260924 16:05:39.015440 10205 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:39.015501 10205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:39.019833 10205 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:05:39.022061 10220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:39.023389 10205 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:05:39.023597 10205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "2e6988e6593c40fbab8f0625315830d3"
format_stamp: "Formatted at 2026-09-24 16:04:56 on dist-test-slave-n106"
server_key: "e70a4b38b417a5725e3cbbd04d0a5d6c"
server_key_iv: "3a59fa88f803bba63b4e1a9b0f2fa4cc"
server_key_version: "encryptionkey@0"
I20260924 16:05:39.024150 10205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:39.036741 10205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:39.039799 10205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:39.040015 10205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:39.047801 10205 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:40819
I20260924 16:05:39.048091 10272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:40819 every 8 connection(s)
I20260924 16:05:39.049062 10205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:05:39.051728 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 10205
I20260924 16:05:39.052232 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:33851
--local_ip_for_outbound_sockets=127.25.109.1
--tserver_master_addrs=127.25.109.62:40819
--webserver_port=44523
--webserver_interface=127.25.109.1
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:05:39.054044 10273 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:05:39.056089 10273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3: Bootstrap starting.
I20260924 16:05:39.074889 10273 log.cc:824] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:39.086740 10273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:05:39.087199 10273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3: Bootstrap complete.
I20260924 16:05:39.092276 10273 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6988e6593c40fbab8f0625315830d3" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40819 } }
I20260924 16:05:39.092833 10273 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e6988e6593c40fbab8f0625315830d3, State: Initialized, Role: FOLLOWER
I20260924 16:05:39.093372 10273 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6988e6593c40fbab8f0625315830d3" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40819 } }
I20260924 16:05:39.093539 10273 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:05:39.093926 10273 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:05:39.094269 10273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [term 2 FOLLOWER]: Advancing to term 3
I20260924 16:05:39.096640 10273 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6988e6593c40fbab8f0625315830d3" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40819 } }
I20260924 16:05:39.097025 10273 leader_election.cc:302] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e6988e6593c40fbab8f0625315830d3; no voters:
I20260924 16:05:39.097411 10273 leader_election.cc:288] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 16:05:39.097536 10277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 16:05:39.097827 10277 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [term 3 LEADER]: Becoming Leader. State: Replica: 2e6988e6593c40fbab8f0625315830d3, State: Running, Role: LEADER
I20260924 16:05:39.098331 10277 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6988e6593c40fbab8f0625315830d3" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40819 } }
I20260924 16:05:39.098959 10273 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:05:39.100041 10278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2e6988e6593c40fbab8f0625315830d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6988e6593c40fbab8f0625315830d3" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40819 } } }
I20260924 16:05:39.100049 10279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e6988e6593c40fbab8f0625315830d3. Latest consensus state: current_term: 3 leader_uuid: "2e6988e6593c40fbab8f0625315830d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6988e6593c40fbab8f0625315830d3" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40819 } } }
I20260924 16:05:39.100229 10278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:39.100230 10279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3 [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:39.104857 10293 catalog_manager.cc:1259] Loaded cluster ID: 5562202b9b744fc0b78d12c52e0503ce
I20260924 16:05:39.104929 10293 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3: loading cluster ID for follower catalog manager: success
I20260924 16:05:39.107888 10293 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2e6988e6593c40fbab8f0625315830d3: acquiring CA information for follower catalog manager: success
F20260924 16:05:39.109323 10293 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7fa14599ad google::LogMessage::Fail() at ??:0
@ 0x7f7fa145c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f7fa14594de google::LogMessage::Flush() at ??:0
@ 0x7f7fa145aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7fa411d18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f7fabdc2989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7f7fabd9f023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7f7fabd9f54b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7f7fabd98652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7f7fabd979e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7f7fabdde541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7fa369c512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7fa4ce06db start_thread at ??:0
@ 0x7f7f9ff5761f clone at ??:0
W20260924 16:05:39.199716 10275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:39.199977 10275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:39.200021 10275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:39.204113 10275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:39.204175 10275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:39.204262 10275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:05:39.209700 10275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:33851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=44523
--tserver_master_addrs=127.25.109.62:40819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:05:39.210837 10275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:39.211934 10275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:39.223213 10314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:39.223163 10317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:39.223272 10315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:39.223598 10275 server_base.cc:1034] running on GCE node
I20260924 16:05:39.224022 10275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:39.224792 10275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:39.225988 10275 hybrid_clock.cc:648] HybridClock initialized: now 1790265939225978 us; error 32 us; skew 500 ppm
Sep 24 16:05:39 dist-test-slave-n106 krb5kdc[9124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265939, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:39.228864 10275 init.cc:377] Logged in from keytab as kudu/127.25.109.1@KRBTEST.COM (short username kudu)
I20260924 16:05:39.229822 10275 webserver.cc:460] Webserver started at http://127.25.109.1:44523/ using document root <none> and password file <none>
I20260924 16:05:39.230428 10275 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:39.230497 10275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:39.234661 10275 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:05:39.237059 10324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:39.238669 10275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:05:39.238791 10275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "3ab79c4bfa714ec89007db708ee1e66f"
format_stamp: "Formatted at 2026-09-24 16:04:57 on dist-test-slave-n106"
server_key: "00c0523734fef87fc5f2c91428e91271"
server_key_iv: "9e2d5ce57e0f97f4a8de9c85802e0d5c"
server_key_version: "encryptionkey@0"
I20260924 16:05:39.239487 10275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:39.258152 10275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:39.262012 10275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:39.262235 10275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:39.262904 10275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:05:39.264073 10331 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 16:05:39.270130 10275 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 16:05:39.270215 10275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260924 16:05:39.270315 10275 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 16:05:39.272045 10275 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 16:05:39.272111 10275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 16:05:39.272511 10331 tablet_bootstrap.cc:492] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f: Bootstrap starting.
I20260924 16:05:39.282778 10275 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:33851
I20260924 16:05:39.282807 10439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:33851 every 8 connection(s)
I20260924 16:05:39.283740 10275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:05:39.291443 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 10275
I20260924 16:05:39.291900 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:38223
--local_ip_for_outbound_sockets=127.25.109.2
--tserver_master_addrs=127.25.109.62:40819
--webserver_port=37343
--webserver_interface=127.25.109.2
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:05:39.296630 10331 log.cc:824] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:39.306716 10331 tablet_bootstrap.cc:492] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:05:39.307191 10331 tablet_bootstrap.cc:492] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f: Bootstrap complete.
I20260924 16:05:39.308830 10331 ts_tablet_manager.cc:1397] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f: Time spent bootstrapping tablet: real 0.037s user 0.022s sys 0.014s
I20260924 16:05:39.312568 10331 raft_consensus.cc:348] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c780642dde34cc9b244869a25223c8b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35851 } } peers { permanent_uuid: "11f08c0f2df24d5388cedd0f60a66f92" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38223 } } peers { permanent_uuid: "3ab79c4bfa714ec89007db708ee1e66f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33851 } }
I20260924 16:05:39.313162 10331 raft_consensus.cc:729] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ab79c4bfa714ec89007db708ee1e66f, State: Initialized, Role: FOLLOWER
I20260924 16:05:39.313736 10331 consensus_queue.cc:260] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c780642dde34cc9b244869a25223c8b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35851 } } peers { permanent_uuid: "11f08c0f2df24d5388cedd0f60a66f92" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38223 } } peers { permanent_uuid: "3ab79c4bfa714ec89007db708ee1e66f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33851 } }
I20260924 16:05:39.314851 10331 ts_tablet_manager.cc:1428] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
W20260924 16:05:39.336891 10442 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:05:39.285662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.1:34317)
0924 16:05:39.286571 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:39.286611 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:39.287348 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:39.335720 (+ 48372us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":736,"thread_start_us":114,"threads_started":1}
W20260924 16:05:39.337469 10440 heartbeater.cc:644] Failed to heartbeat to 127.25.109.62:40819 (0 consecutive failures): Network error: Failed to ping master at 127.25.109.62:40819: Client connection negotiation failed: client connection to 127.25.109.62:40819: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 16:05:39.338268 10442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:39.338014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.1:37435)
0924 16:05:39.338147 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:39.338212 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 16:05:39.338956 10442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:39.338731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.1:59359)
0924 16:05:39.338848 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:39.338906 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 16:05:39.339134 10440 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 16:05:39.406337 10443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:39.406683 10443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:39.406759 10443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:39.410836 10443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:39.410933 10443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:39.411026 10443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:05:39.416314 10443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:38223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=37343
--tserver_master_addrs=127.25.109.62:40819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:05:39.417438 10443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:39.418509 10443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:39.428234 10453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:39.428205 10450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:39.428224 10451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:39.428397 10443 server_base.cc:1034] running on GCE node
I20260924 16:05:39.429008 10443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:39.429642 10443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:39.430804 10443 hybrid_clock.cc:648] HybridClock initialized: now 1790265939430756 us; error 60 us; skew 500 ppm
Sep 24 16:05:39 dist-test-slave-n106 krb5kdc[9124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265939, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:39.433605 10443 init.cc:377] Logged in from keytab as kudu/127.25.109.2@KRBTEST.COM (short username kudu)
I20260924 16:05:39.434432 10443 webserver.cc:460] Webserver started at http://127.25.109.2:37343/ using document root <none> and password file <none>
I20260924 16:05:39.434974 10443 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:39.435032 10443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:39.438984 10443 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 16:05:39.441004 10460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:39.442235 10443 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:05:39.442335 10443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "11f08c0f2df24d5388cedd0f60a66f92"
format_stamp: "Formatted at 2026-09-24 16:04:57 on dist-test-slave-n106"
server_key: "6f297cfe1d5d9fe15aafc2974ebc4387"
server_key_iv: "59c983a953a0137da7154db4c6f280fe"
server_key_version: "encryptionkey@0"
I20260924 16:05:39.442863 10443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:39.477130 10443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:39.480237 10443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:39.480399 10443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:39.480980 10443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:05:39.482053 10467 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260924 16:05:39.486039 10467 ts_tablet_manager.cc:1852] T 7f06daada7c744d78b792be13e8d2635 P 11f08c0f2df24d5388cedd0f60a66f92: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 16:05:39.486454 10467 ts_tablet_manager.cc:1905] T 7f06daada7c744d78b792be13e8d2635 P 11f08c0f2df24d5388cedd0f60a66f92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:05:39.490583 10467 ts_tablet_manager.cc:1918] T 7f06daada7c744d78b792be13e8d2635 P 11f08c0f2df24d5388cedd0f60a66f92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:05:39.490783 10467 ts_tablet_manager.cc:1939] T 7f06daada7c744d78b792be13e8d2635 P 11f08c0f2df24d5388cedd0f60a66f92: Deleting consensus metadata
I20260924 16:05:39.491472 10443 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 16:05:39.491530 10443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.000s
I20260924 16:05:39.491609 10443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:39.491652 10443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:39.500703 10443 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:38223
I20260924 16:05:39.500762 10575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:38223 every 8 connection(s)
I20260924 16:05:39.501616 10443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:05:39.505136 10578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:39.503756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.2:42463)
0924 16:05:39.504251 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:39.504365 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":117,"threads_started":1}
W20260924 16:05:39.505564 10576 heartbeater.cc:644] Failed to heartbeat to 127.25.109.62:40819 (0 consecutive failures): Network error: Failed to ping master at 127.25.109.62:40819: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
I20260924 16:05:39.506110 10578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:39.505900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.2:49927)
0924 16:05:39.506009 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:39.506056 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 16:05:39.506672 10578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:39.506486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.2:48891)
0924 16:05:39.506580 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:39.506622 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 16:05:39.506839 10576 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 16:05:39.509076 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 10443
I20260924 16:05:39.509518 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:35851
--local_ip_for_outbound_sockets=127.25.109.3
--tserver_master_addrs=127.25.109.62:40819
--webserver_port=34043
--webserver_interface=127.25.109.3
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:39.621357 10579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:39.621603 10579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:39.621644 10579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:39.625730 10579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:39.625782 10579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:39.625873 10579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:05:39.631078 10579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:35851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=34043
--tserver_master_addrs=127.25.109.62:40819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:05:39.632072 10579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:39.633129 10579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:39.642927 10587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:39.643014 10579 server_base.cc:1034] running on GCE node
W20260924 16:05:39.642927 10585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:39.642933 10584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:39.643635 10579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:39.644318 10579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:39.645573 10579 hybrid_clock.cc:648] HybridClock initialized: now 1790265939645536 us; error 46 us; skew 500 ppm
Sep 24 16:05:39 dist-test-slave-n106 krb5kdc[9124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265939, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:39.648360 10579 init.cc:377] Logged in from keytab as kudu/127.25.109.3@KRBTEST.COM (short username kudu)
I20260924 16:05:39.649259 10579 webserver.cc:460] Webserver started at http://127.25.109.3:34043/ using document root <none> and password file <none>
I20260924 16:05:39.649783 10579 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:39.649840 10579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:39.653486 10579 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260924 16:05:39.655426 10594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:39.656683 10579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260924 16:05:39.656818 10579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "3c780642dde34cc9b244869a25223c8b"
format_stamp: "Formatted at 2026-09-24 16:04:57 on dist-test-slave-n106"
server_key: "7d8b653ade785dfeeaac4f05d9bcbac3"
server_key_iv: "39f7befb2b91d0ce6f5bcab435f4e500"
server_key_version: "encryptionkey@0"
I20260924 16:05:39.657347 10579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:39.668623 10579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:39.671253 10579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:39.671449 10579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:39.672009 10579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:05:39.672884 10579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:39.672956 10579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:39.673023 10579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:39.673079 10579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:39.681836 10579 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:35851
I20260924 16:05:39.681859 10709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:35851 every 8 connection(s)
I20260924 16:05:39.682812 10579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:05:39.685738 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 10579
I20260924 16:05:39.686823 10712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:39.684926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.3:56287)
0924 16:05:39.685678 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:39.685856 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":108,"threads_started":1}
W20260924 16:05:39.687286 10710 heartbeater.cc:644] Failed to heartbeat to 127.25.109.62:40819 (0 consecutive failures): Network error: Failed to ping master at 127.25.109.62:40819: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
I20260924 16:05:39.687858 10712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:39.687644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.3:42909)
0924 16:05:39.687760 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:39.687806 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 16:05:39.688657 10712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:39.688447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.3:58141)
0924 16:05:39.688551 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:39.688592 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 16:05:39.688881 10710 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 16:05:40.340149 10714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:40.339712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.1:53055)
0924 16:05:40.339976 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:40.340032 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":72,"threads_started":1}
I20260924 16:05:40.507829 10715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:40.507457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.2:35821)
0924 16:05:40.507678 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:40.507733 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":64,"threads_started":1}
I20260924 16:05:40.689795 10716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:40.689416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.3:40093)
0924 16:05:40.689644 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:40.689698 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":65,"threads_started":1}
I20260924 16:05:40.780643 10717 raft_consensus.cc:482] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:05:40.780897 10717 raft_consensus.cc:504] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c780642dde34cc9b244869a25223c8b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35851 } } peers { permanent_uuid: "11f08c0f2df24d5388cedd0f60a66f92" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38223 } } peers { permanent_uuid: "3ab79c4bfa714ec89007db708ee1e66f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33851 } }
I20260924 16:05:40.782078 10717 leader_election.cc:288] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c780642dde34cc9b244869a25223c8b (127.25.109.3:35851), 11f08c0f2df24d5388cedd0f60a66f92 (127.25.109.2:38223)
Sep 24 16:05:40 dist-test-slave-n106 krb5kdc[9124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265939, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for kudu/127.25.109.2@KRBTEST.COM
Sep 24 16:05:40 dist-test-slave-n106 krb5kdc[9124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265939, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for kudu/127.25.109.3@KRBTEST.COM
I20260924 16:05:40.798444 10718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:40.782344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:35851 (local address 127.25.109.1:54385)
0924 16:05:40.782821 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:40.782834 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:40.782929 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:40.784692 (+ 1763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:40.784698 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:40.785234 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:40.785744 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:40.785772 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:40.787097 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:40.787100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:40.788751 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:40.788763 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:40.789724 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:40.789728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:40.790059 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:40.790844 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:40.790873 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:40.793051 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:40.795309 (+ 2258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:40.795316 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:40.795331 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:40.795854 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:40.796259 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:40.796263 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:40.796265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:40.796521 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:40.797273 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:40.797279 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:40.797668 (+ 389us) client_negotiation.cc:769] Sending connection context
0924 16:05:40.797880 (+ 212us) client_negotiation.cc:240] Negotiation successful
0924 16:05:40.798285 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":626,"spinlock_wait_cycles":83456,"thread_start_us":86,"threads_started":1}
I20260924 16:05:40.798445 10714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:40.782344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:38223 (local address 127.25.109.1:46013)
0924 16:05:40.782444 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:40.782460 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:40.782573 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:40.784692 (+ 2119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:40.784698 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:40.785233 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:40.785745 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:40.785772 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:40.787058 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:40.787064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:40.788751 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:40.788763 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:40.789724 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:40.789728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:40.790062 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:40.790844 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:40.790871 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:40.792775 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:40.795309 (+ 2534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:40.795316 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:40.795331 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:40.795854 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:40.796260 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:40.796263 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:40.796265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:40.796521 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:40.797272 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:40.797279 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:40.797668 (+ 389us) client_negotiation.cc:769] Sending connection context
0924 16:05:40.797880 (+ 212us) client_negotiation.cc:240] Negotiation successful
0924 16:05:40.798234 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":393,"spinlock_wait_cycles":63872}
I20260924 16:05:40.798506 10719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:40.782553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:46013 (local address 127.25.109.2:38223)
0924 16:05:40.782883 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:40.782902 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:40.782947 (+ 45us) server_negotiation.cc:407] Connection header received
0924 16:05:40.783872 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:40.783882 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:40.784273 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:40.784642 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:40.785951 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:40.786916 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:40.788885 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:40.789585 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:40.789766 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:40.793017 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:40.793022 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:40.793034 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:40.793061 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:40.795169 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:40.795978 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:40.795985 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:40.795991 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:40.796157 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:40.796624 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:40.796627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:40.796628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:40.797153 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:40.797285 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:40.798036 (+ 751us) server_negotiation.cc:299] Negotiation successful
0924 16:05:40.798322 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":82,"threads_started":1}
I20260924 16:05:40.799377 10720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:40.782576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:54385 (local address 127.25.109.3:35851)
0924 16:05:40.782959 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:40.782978 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:40.783014 (+ 36us) server_negotiation.cc:407] Connection header received
0924 16:05:40.783821 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:40.783830 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:40.784272 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:40.784642 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:40.785946 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:40.786974 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:40.788885 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:40.789586 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:40.789767 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:40.793228 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:40.793234 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:40.793249 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:40.793282 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:40.795196 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:40.795978 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:40.795985 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:40.795991 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:40.796155 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:40.796630 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:40.796635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:40.796636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:40.797146 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:40.797285 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:40.798036 (+ 751us) server_negotiation.cc:299] Negotiation successful
0924 16:05:40.798322 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":112,"threads_started":1}
W20260924 16:05:40.800941 10328 leader_election.cc:341] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 11f08c0f2df24d5388cedd0f60a66f92 (127.25.109.2:38223): Not found: Tablet not found: 7f06daada7c744d78b792be13e8d2635
W20260924 16:05:40.801379 10327 leader_election.cc:341] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c780642dde34cc9b244869a25223c8b (127.25.109.3:35851): Not found: Tablet not found: 7f06daada7c744d78b792be13e8d2635
I20260924 16:05:40.801473 10327 leader_election.cc:302] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ab79c4bfa714ec89007db708ee1e66f; no voters: 11f08c0f2df24d5388cedd0f60a66f92, 3c780642dde34cc9b244869a25223c8b
I20260924 16:05:40.801744 10717 raft_consensus.cc:2726] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:05:41.341339 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:41.340928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.1:45747)
0924 16:05:41.341185 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:41.341243 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":79,"threads_started":1}
I20260924 16:05:41.509039 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:41.508654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.2:38767)
0924 16:05:41.508891 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:41.508948 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":70,"threads_started":1}
I20260924 16:05:41.690804 10723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:41.690441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.3:34743)
0924 16:05:41.690670 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:41.690720 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":64,"threads_started":1}
I20260924 16:05:42.342483 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:42.342053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.1:44667)
0924 16:05:42.342328 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:42.342386 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":76,"threads_started":1}
I20260924 16:05:42.510135 10725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:42.509760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.2:59057)
0924 16:05:42.509991 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:42.510048 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":72,"threads_started":1}
I20260924 16:05:42.691855 10726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:42.691475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.3:40231)
0924 16:05:42.691711 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:42.691767 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":71,"threads_started":1}
I20260924 16:05:42.892620 10727 raft_consensus.cc:482] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:05:42.892779 10727 raft_consensus.cc:504] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c780642dde34cc9b244869a25223c8b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35851 } } peers { permanent_uuid: "11f08c0f2df24d5388cedd0f60a66f92" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38223 } } peers { permanent_uuid: "3ab79c4bfa714ec89007db708ee1e66f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33851 } }
I20260924 16:05:42.893226 10727 leader_election.cc:288] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c780642dde34cc9b244869a25223c8b (127.25.109.3:35851), 11f08c0f2df24d5388cedd0f60a66f92 (127.25.109.2:38223)
W20260924 16:05:42.894112 10327 leader_election.cc:341] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c780642dde34cc9b244869a25223c8b (127.25.109.3:35851): Not found: Tablet not found: 7f06daada7c744d78b792be13e8d2635
W20260924 16:05:42.894263 10328 leader_election.cc:341] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 11f08c0f2df24d5388cedd0f60a66f92 (127.25.109.2:38223): Not found: Tablet not found: 7f06daada7c744d78b792be13e8d2635
I20260924 16:05:42.894346 10328 leader_election.cc:302] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ab79c4bfa714ec89007db708ee1e66f; no voters: 11f08c0f2df24d5388cedd0f60a66f92, 3c780642dde34cc9b244869a25223c8b
I20260924 16:05:42.894481 10727 raft_consensus.cc:2726] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:05:43.343652 10728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:43.343242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.1:48537)
0924 16:05:43.343497 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:43.343554 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":76,"threads_started":1}
I20260924 16:05:43.511371 10729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:43.510934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.2:48923)
0924 16:05:43.511215 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:43.511266 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":124,"threads_started":1}
I20260924 16:05:43.692932 10730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:43.692544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.3:35641)
0924 16:05:43.692794 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:43.692849 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":79,"threads_started":1}
I20260924 16:05:44.345155 10731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:44.344626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.1:40109)
0924 16:05:44.344979 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:44.345042 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":96,"threads_started":1}
I20260924 16:05:44.512568 10732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:44.512116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.2:56103)
0924 16:05:44.512366 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:44.512470 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":75,"threads_started":1}
I20260924 16:05:44.694307 10733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:44.693836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.3:33243)
0924 16:05:44.694148 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:44.694207 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":119,"threads_started":1}
I20260924 16:05:45.318776 10734 raft_consensus.cc:482] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:05:45.318917 10734 raft_consensus.cc:504] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c780642dde34cc9b244869a25223c8b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35851 } } peers { permanent_uuid: "11f08c0f2df24d5388cedd0f60a66f92" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38223 } } peers { permanent_uuid: "3ab79c4bfa714ec89007db708ee1e66f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33851 } }
I20260924 16:05:45.319346 10734 leader_election.cc:288] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c780642dde34cc9b244869a25223c8b (127.25.109.3:35851), 11f08c0f2df24d5388cedd0f60a66f92 (127.25.109.2:38223)
W20260924 16:05:45.320067 10327 leader_election.cc:341] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c780642dde34cc9b244869a25223c8b (127.25.109.3:35851): Not found: Tablet not found: 7f06daada7c744d78b792be13e8d2635
W20260924 16:05:45.320159 10328 leader_election.cc:341] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 11f08c0f2df24d5388cedd0f60a66f92 (127.25.109.2:38223): Not found: Tablet not found: 7f06daada7c744d78b792be13e8d2635
I20260924 16:05:45.320269 10328 leader_election.cc:302] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ab79c4bfa714ec89007db708ee1e66f; no voters: 11f08c0f2df24d5388cedd0f60a66f92, 3c780642dde34cc9b244869a25223c8b
I20260924 16:05:45.320472 10734 raft_consensus.cc:2726] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:05:45.346392 10735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:45.345885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.1:54251)
0924 16:05:45.346188 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:45.346253 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":73,"threads_started":1}
I20260924 16:05:45.513635 10736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:45.513241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.2:38799)
0924 16:05:45.513474 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:45.513529 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":68,"threads_started":1}
I20260924 16:05:45.695567 10737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:45.695130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.3:49159)
0924 16:05:45.695407 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:45.695480 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":71,"threads_started":1}
I20260924 16:05:46.347502 10738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:46.347119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.1:53409)
0924 16:05:46.347349 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:46.347403 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":73,"threads_started":1}
I20260924 16:05:46.514871 10739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:46.514371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.2:48565)
0924 16:05:46.514714 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:46.514775 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":74,"threads_started":1}
I20260924 16:05:46.696774 10740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:46.696283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.3:44617)
0924 16:05:46.696626 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:46.696680 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":116,"threads_started":1}
I20260924 16:05:47.348765 10741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:47.348271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.1:50713)
0924 16:05:47.348592 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:47.348665 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":77,"threads_started":1}
I20260924 16:05:47.516103 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:47.515701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.2:42781)
0924 16:05:47.515942 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:47.516009 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":67,"threads_started":1}
I20260924 16:05:47.697877 10743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:47.697493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.3:49627)
0924 16:05:47.697735 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:47.697786 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":69,"threads_started":1}
I20260924 16:05:48.350011 10744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:48.349582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.1:44801)
0924 16:05:48.349858 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:48.349914 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":92,"threads_started":1}
I20260924 16:05:48.517390 10745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:48.516943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.2:45699)
0924 16:05:48.517221 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:48.517287 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":85,"threads_started":1}
I20260924 16:05:48.699025 10746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:48.698624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.3:53499)
0924 16:05:48.698875 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:48.698932 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":84,"threads_started":1}
I20260924 16:05:49.351255 10747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:49.350855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.1:55325)
0924 16:05:49.351100 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:49.351158 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":75,"threads_started":1}
I20260924 16:05:49.518678 10748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:49.518236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.2:54167)
0924 16:05:49.518497 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:49.518558 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":78,"threads_started":1}
I20260924 16:05:49.700238 10749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:49.699816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.3:35325)
0924 16:05:49.700072 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:49.700128 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":72,"threads_started":1}
I20260924 16:05:50.213125 10750 raft_consensus.cc:482] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:05:50.213255 10750 raft_consensus.cc:504] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c780642dde34cc9b244869a25223c8b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35851 } } peers { permanent_uuid: "11f08c0f2df24d5388cedd0f60a66f92" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38223 } } peers { permanent_uuid: "3ab79c4bfa714ec89007db708ee1e66f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33851 } }
I20260924 16:05:50.213714 10750 leader_election.cc:288] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c780642dde34cc9b244869a25223c8b (127.25.109.3:35851), 11f08c0f2df24d5388cedd0f60a66f92 (127.25.109.2:38223)
W20260924 16:05:50.214526 10328 leader_election.cc:341] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 11f08c0f2df24d5388cedd0f60a66f92 (127.25.109.2:38223): Not found: Tablet not found: 7f06daada7c744d78b792be13e8d2635
W20260924 16:05:50.214661 10327 leader_election.cc:341] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c780642dde34cc9b244869a25223c8b (127.25.109.3:35851): Not found: Tablet not found: 7f06daada7c744d78b792be13e8d2635
I20260924 16:05:50.214756 10327 leader_election.cc:302] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ab79c4bfa714ec89007db708ee1e66f; no voters: 11f08c0f2df24d5388cedd0f60a66f92, 3c780642dde34cc9b244869a25223c8b
I20260924 16:05:50.214937 10750 raft_consensus.cc:2726] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:05:50.352506 10751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:50.352016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.1:48903)
0924 16:05:50.352279 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:50.352341 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":76,"threads_started":1}
I20260924 16:05:50.519812 10752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:50.519414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.2:53129)
0924 16:05:50.519665 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:50.519718 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":85,"threads_started":1}
I20260924 16:05:50.701397 10753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:50.701007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.3:44827)
0924 16:05:50.701247 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:50.701302 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":70,"threads_started":1}
I20260924 16:05:51.353681 10754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:51.353281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.1:58591)
0924 16:05:51.353532 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:51.353586 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":78,"threads_started":1}
I20260924 16:05:51.521279 10755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:51.520841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.2:47849)
0924 16:05:51.521110 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:51.521177 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":102,"threads_started":1}
I20260924 16:05:51.702610 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:51.702156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.3:41499)
0924 16:05:51.702439 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:51.702503 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":73,"threads_started":1}
I20260924 16:05:52.354827 10757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:52.354386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.1:47833)
0924 16:05:52.354671 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:52.354728 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":74,"threads_started":1}
I20260924 16:05:52.522583 10758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:52.522122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.2:55785)
0924 16:05:52.522424 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:52.522484 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":88,"threads_started":1}
I20260924 16:05:52.703827 10759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:52.703422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.3:44487)
0924 16:05:52.703667 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:52.703727 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":68,"threads_started":1}
I20260924 16:05:53.356176 10760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:53.355682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.1:33999)
0924 16:05:53.355965 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:53.356051 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":113,"threads_started":1}
I20260924 16:05:53.523900 10761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:53.523450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.2:46775)
0924 16:05:53.523740 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:53.523798 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":80,"threads_started":1}
I20260924 16:05:53.607057 10762 raft_consensus.cc:482] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:05:53.607188 10762 raft_consensus.cc:504] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c780642dde34cc9b244869a25223c8b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 35851 } } peers { permanent_uuid: "11f08c0f2df24d5388cedd0f60a66f92" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 38223 } } peers { permanent_uuid: "3ab79c4bfa714ec89007db708ee1e66f" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 33851 } }
I20260924 16:05:53.607671 10762 leader_election.cc:288] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c780642dde34cc9b244869a25223c8b (127.25.109.3:35851), 11f08c0f2df24d5388cedd0f60a66f92 (127.25.109.2:38223)
W20260924 16:05:53.608536 10327 leader_election.cc:341] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c780642dde34cc9b244869a25223c8b (127.25.109.3:35851): Not found: Tablet not found: 7f06daada7c744d78b792be13e8d2635
W20260924 16:05:53.608641 10328 leader_election.cc:341] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 11f08c0f2df24d5388cedd0f60a66f92 (127.25.109.2:38223): Not found: Tablet not found: 7f06daada7c744d78b792be13e8d2635
I20260924 16:05:53.608750 10328 leader_election.cc:302] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ab79c4bfa714ec89007db708ee1e66f; no voters: 11f08c0f2df24d5388cedd0f60a66f92, 3c780642dde34cc9b244869a25223c8b
I20260924 16:05:53.608897 10762 raft_consensus.cc:2726] T 7f06daada7c744d78b792be13e8d2635 P 3ab79c4bfa714ec89007db708ee1e66f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:05:53.705127 10763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:53.704685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.3:48465)
0924 16:05:53.704953 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:53.705015 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":95,"threads_started":1}
I20260924 16:05:54.357594 10764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:54.357172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.1:46347)
0924 16:05:54.357424 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:54.357498 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":85,"threads_started":1}
I20260924 16:05:54.525288 10765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:54.524899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40819 (local address 127.25.109.2:35121)
0924 16:05:54.525138 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:54.525191 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.62:40819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":68,"threads_started":1}
I20260924 16:05:54.687664 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 10275
I20260924 16:05:54.694022 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 10443
I20260924 16:05:54.699612 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 10579
2026-09-24T16:05:54Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59160 ms)
[----------] 30 tests from SecurityITest (245580 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:05:54 dist-test-slave-n106 krb5kdc[10768](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:05:54 dist-test-slave-n106 krb5kdc[10768](info): set up 2 sockets
Sep 24 16:05:54 dist-test-slave-n106 krb5kdc[10768](info): commencing operation
krb5kdc: starting...
W20260924 16:05:55.745409 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:05:55 dist-test-slave-n106 krb5kdc[10768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:05:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:05:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
I20260924 16:05:55.866853 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:33269
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:33609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:33269
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:55.977125 10781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:55.977433 10781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:55.977520 10781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:55.981575 10781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:05:55.981667 10781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:55.981705 10781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:55.981757 10781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:05:55.981793 10781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:05:55.987094 10781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:33609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:33269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:33269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:05:55.988260 10781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:55.989413 10781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:55.998378 10787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:55.998437 10790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:55.998361 10788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:55.998515 10781 server_base.cc:1034] running on GCE node
I20260924 16:05:55.999246 10781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:56.000375 10781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:56.001623 10781 hybrid_clock.cc:648] HybridClock initialized: now 1790265956001604 us; error 32 us; skew 500 ppm
Sep 24 16:05:55 dist-test-slave-n106 krb5kdc[10768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265955, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:56.004380 10781 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:05:56.005304 10781 webserver.cc:460] Webserver started at http://127.25.109.62:46093/ using document root <none> and password file <none>
I20260924 16:05:56.005879 10781 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:56.005971 10781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:56.006207 10781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:56.008211 10781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "e80cc00a493d4fe8b698dd383d6cead2"
format_stamp: "Formatted at 2026-09-24 16:05:56 on dist-test-slave-n106"
server_key: "b206514924646ac1062580393656ed80"
server_key_iv: "3fe0a7b0b4cfcffa3ba3a9517a908e3a"
server_key_version: "encryptionkey@0"
I20260924 16:05:56.008826 10781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "e80cc00a493d4fe8b698dd383d6cead2"
format_stamp: "Formatted at 2026-09-24 16:05:56 on dist-test-slave-n106"
server_key: "b206514924646ac1062580393656ed80"
server_key_iv: "3fe0a7b0b4cfcffa3ba3a9517a908e3a"
server_key_version: "encryptionkey@0"
I20260924 16:05:56.012508 10781 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 16:05:56.014925 10797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:56.016141 10781 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:05:56.016278 10781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "e80cc00a493d4fe8b698dd383d6cead2"
format_stamp: "Formatted at 2026-09-24 16:05:56 on dist-test-slave-n106"
server_key: "b206514924646ac1062580393656ed80"
server_key_iv: "3fe0a7b0b4cfcffa3ba3a9517a908e3a"
server_key_version: "encryptionkey@0"
I20260924 16:05:56.016391 10781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:56.036535 10781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:56.039479 10781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:56.039700 10781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:56.046775 10781 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:33269
I20260924 16:05:56.046837 10849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:33269 every 8 connection(s)
I20260924 16:05:56.047760 10781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:05:56.050589 10850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:56.052788 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 10781
I20260924 16:05:56.052946 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:05:56.053251 26036 external_mini_cluster.cc:1428] Setting key 982c7b630e4e40eb2c0faa131c7cc7aa
I20260924 16:05:56.056914 10850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e80cc00a493d4fe8b698dd383d6cead2: Bootstrap starting.
I20260924 16:05:56.059509 10850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e80cc00a493d4fe8b698dd383d6cead2: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:56.060424 10850 log.cc:824] T 00000000000000000000000000000000 P e80cc00a493d4fe8b698dd383d6cead2: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:56.062577 10850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e80cc00a493d4fe8b698dd383d6cead2: No bootstrap required, opened a new log
Sep 24 16:05:56 dist-test-slave-n106 krb5kdc[10768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:05:56.066882 10850 raft_consensus.cc:348] T 00000000000000000000000000000000 P e80cc00a493d4fe8b698dd383d6cead2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80cc00a493d4fe8b698dd383d6cead2" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33269 } }
I20260924 16:05:56.067183 10850 raft_consensus.cc:374] T 00000000000000000000000000000000 P e80cc00a493d4fe8b698dd383d6cead2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:56.067243 10850 raft_consensus.cc:729] T 00000000000000000000000000000000 P e80cc00a493d4fe8b698dd383d6cead2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e80cc00a493d4fe8b698dd383d6cead2, State: Initialized, Role: FOLLOWER
I20260924 16:05:56.067750 10850 consensus_queue.cc:260] T 00000000000000000000000000000000 P e80cc00a493d4fe8b698dd383d6cead2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80cc00a493d4fe8b698dd383d6cead2" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33269 } }
I20260924 16:05:56.067914 10850 raft_consensus.cc:388] T 00000000000000000000000000000000 P e80cc00a493d4fe8b698dd383d6cead2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:05:56.068017 10850 raft_consensus.cc:482] T 00000000000000000000000000000000 P e80cc00a493d4fe8b698dd383d6cead2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:05:56.068125 10850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e80cc00a493d4fe8b698dd383d6cead2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:56.069205 10853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:56.054648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43706 (local address 127.25.109.62:33269)
0924 16:05:56.055064 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:56.055084 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:56.055128 (+ 44us) server_negotiation.cc:407] Connection header received
0924 16:05:56.055942 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:56.055967 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:56.056368 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:56.056764 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:56.057475 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.058977 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.059889 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.060619 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.060797 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:56.063428 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:56.063434 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:56.063450 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:56.063480 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:56.066036 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:56.066457 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:56.066464 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:56.066470 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:56.066616 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:56.066871 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:56.066874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:56.066876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:56.067406 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:56.067605 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:56.067884 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 16:05:56.068191 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":103,"threads_started":1}
I20260924 16:05:56.069231 10850 raft_consensus.cc:504] T 00000000000000000000000000000000 P e80cc00a493d4fe8b698dd383d6cead2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80cc00a493d4fe8b698dd383d6cead2" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33269 } }
I20260924 16:05:56.069571 10850 leader_election.cc:302] T 00000000000000000000000000000000 P e80cc00a493d4fe8b698dd383d6cead2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e80cc00a493d4fe8b698dd383d6cead2; no voters:
I20260924 16:05:56.069886 10850 leader_election.cc:288] T 00000000000000000000000000000000 P e80cc00a493d4fe8b698dd383d6cead2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:05:56.070006 10855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e80cc00a493d4fe8b698dd383d6cead2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:05:56.070215 10855 raft_consensus.cc:686] T 00000000000000000000000000000000 P e80cc00a493d4fe8b698dd383d6cead2 [term 1 LEADER]: Becoming Leader. State: Replica: e80cc00a493d4fe8b698dd383d6cead2, State: Running, Role: LEADER
I20260924 16:05:56.070596 10855 consensus_queue.cc:237] T 00000000000000000000000000000000 P e80cc00a493d4fe8b698dd383d6cead2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80cc00a493d4fe8b698dd383d6cead2" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33269 } }
I20260924 16:05:56.071115 10850 sys_catalog.cc:564] T 00000000000000000000000000000000 P e80cc00a493d4fe8b698dd383d6cead2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:05:56.071777 10857 sys_catalog.cc:455] T 00000000000000000000000000000000 P e80cc00a493d4fe8b698dd383d6cead2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e80cc00a493d4fe8b698dd383d6cead2. Latest consensus state: current_term: 1 leader_uuid: "e80cc00a493d4fe8b698dd383d6cead2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80cc00a493d4fe8b698dd383d6cead2" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33269 } } }
I20260924 16:05:56.071893 10857 sys_catalog.cc:458] T 00000000000000000000000000000000 P e80cc00a493d4fe8b698dd383d6cead2 [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:56.072623 10856 sys_catalog.cc:455] T 00000000000000000000000000000000 P e80cc00a493d4fe8b698dd383d6cead2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e80cc00a493d4fe8b698dd383d6cead2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80cc00a493d4fe8b698dd383d6cead2" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 33269 } } }
I20260924 16:05:56.072754 10856 sys_catalog.cc:458] T 00000000000000000000000000000000 P e80cc00a493d4fe8b698dd383d6cead2 [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:56.072667 10861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:05:56.075912 10861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:05:56.081393 10861 catalog_manager.cc:1347] Generated new cluster ID: 9395d49bdae944a293ab457787d8ea13
I20260924 16:05:56.081467 10861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:05:56.102001 10861 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:05:56.103284 10861 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:05:56.114435 10861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e80cc00a493d4fe8b698dd383d6cead2: Generated new TSK 0
I20260924 16:05:56.115252 10861 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
I20260924 16:05:56.147294 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:0
--local_ip_for_outbound_sockets=127.25.109.1
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:33269
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:33609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:56.261143 10876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:56.261384 10876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:56.261421 10876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:56.265576 10876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:56.265635 10876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:56.265718 10876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:05:56.270836 10876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:33609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:33269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:05:56.272027 10876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:56.273173 10876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:56.282954 10881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:56.283030 10876 server_base.cc:1034] running on GCE node
W20260924 16:05:56.282989 10882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:56.282977 10884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:56.283684 10876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:56.284304 10876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:56.285485 10876 hybrid_clock.cc:648] HybridClock initialized: now 1790265956285452 us; error 47 us; skew 500 ppm
Sep 24 16:05:56 dist-test-slave-n106 krb5kdc[10768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265956, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:56.288106 10876 init.cc:377] Logged in from keytab as kudu/127.25.109.1@KRBTEST.COM (short username kudu)
I20260924 16:05:56.289004 10876 webserver.cc:460] Webserver started at http://127.25.109.1:36383/ using document root <none> and password file <none>
I20260924 16:05:56.289518 10876 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:56.289575 10876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:56.289744 10876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:56.291723 10876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-0/data/instance:
uuid: "92b66c114e0e46b29f859f7478567499"
format_stamp: "Formatted at 2026-09-24 16:05:56 on dist-test-slave-n106"
server_key: "974bba092539f8b84a703755cd73f61b"
server_key_iv: "ccaee0718e9bdf7758dc4c34fd3e733d"
server_key_version: "encryptionkey@0"
I20260924 16:05:56.292222 10876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance:
uuid: "92b66c114e0e46b29f859f7478567499"
format_stamp: "Formatted at 2026-09-24 16:05:56 on dist-test-slave-n106"
server_key: "974bba092539f8b84a703755cd73f61b"
server_key_iv: "ccaee0718e9bdf7758dc4c34fd3e733d"
server_key_version: "encryptionkey@0"
I20260924 16:05:56.295792 10876 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:05:56.298563 10891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:56.299724 10876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:05:56.299844 10876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "92b66c114e0e46b29f859f7478567499"
format_stamp: "Formatted at 2026-09-24 16:05:56 on dist-test-slave-n106"
server_key: "974bba092539f8b84a703755cd73f61b"
server_key_iv: "ccaee0718e9bdf7758dc4c34fd3e733d"
server_key_version: "encryptionkey@0"
I20260924 16:05:56.299953 10876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:56.319551 10876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:56.322643 10876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:56.322844 10876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:56.323449 10876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:05:56.324288 10876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:56.324362 10876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:56.324457 10876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:56.324512 10876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:56.333372 10876 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:36043
I20260924 16:05:56.333415 11005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:36043 every 8 connection(s)
I20260924 16:05:56.334306 10876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
Sep 24 16:05:56 dist-test-slave-n106 krb5kdc[10768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265956, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:05:56.343565 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 10876
I20260924 16:05:56.343735 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance
I20260924 16:05:56.344069 26036 external_mini_cluster.cc:1428] Setting key bd6190230f13d292605a1d7fe759dc31
I20260924 16:05:56.348529 10853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:56.336195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:36999 (local address 127.25.109.62:33269)
0924 16:05:56.336353 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:56.336358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:56.337248 (+ 890us) server_negotiation.cc:407] Connection header received
0924 16:05:56.338484 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:56.338487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:56.338534 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:56.338608 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:56.339760 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.340278 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.341435 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.341659 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.341738 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:56.344262 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:56.344267 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:56.344269 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:56.344301 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:56.346119 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:56.346664 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:56.346671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:56.346676 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:56.346778 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:56.347118 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:56.347124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:56.347129 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:56.347396 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:56.347504 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:56.348224 (+ 720us) server_negotiation.cc:299] Negotiation successful
0924 16:05:56.348335 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 16:05:56.349782 11008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:56.336641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33269 (local address 127.25.109.1:36999)
0924 16:05:56.337089 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:56.337133 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:56.338285 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:56.338789 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:56.338796 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:56.339142 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:56.339629 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:56.339643 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.340384 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.340386 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:56.341324 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:56.341334 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.341794 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.341796 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:56.341934 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:56.342466 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:56.342482 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:56.344026 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:56.346245 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:56.346251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:56.346264 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:56.346535 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:56.346874 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:56.346876 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:56.346878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:56.347023 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:56.347544 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:56.347553 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:56.347943 (+ 390us) client_negotiation.cc:769] Sending connection context
0924 16:05:56.348181 (+ 238us) client_negotiation.cc:240] Negotiation successful
0924 16:05:56.348619 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":105,"threads_started":1}
I20260924 16:05:56.351197 11006 heartbeater.cc:347] Connected to a master server at 127.25.109.62:33269
I20260924 16:05:56.351485 11006 heartbeater.cc:464] Registering TS with master...
I20260924 16:05:56.352170 11006 heartbeater.cc:511] Master 127.25.109.62:33269 requested a full tablet report, sending...
I20260924 16:05:56.353905 10814 ts_manager.cc:194] Registered new tserver with Master: 92b66c114e0e46b29f859f7478567499 (127.25.109.1:36043)
WARNING: no policy specified for kudu/127.25.109.2@KRBTEST.COM; defaulting to no policy
I20260924 16:05:56.355140 10814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.1@KRBTEST.COM'} at 127.25.109.1:36999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
I20260924 16:05:56.373539 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:0
--local_ip_for_outbound_sockets=127.25.109.2
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:33269
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:33609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:56.488593 11011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:56.488897 11011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:56.488977 11011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:56.493057 11011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:56.493160 11011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:56.493256 11011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:05:56.498709 11011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:33609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:33269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:05:56.499825 11011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:56.500926 11011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:56.510936 11017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:56.510905 11019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:56.511040 11011 server_base.cc:1034] running on GCE node
W20260924 16:05:56.510910 11016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:56.511695 11011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:56.512356 11011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:56.513569 11011 hybrid_clock.cc:648] HybridClock initialized: now 1790265956513545 us; error 35 us; skew 500 ppm
Sep 24 16:05:56 dist-test-slave-n106 krb5kdc[10768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265956, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:56.516534 11011 init.cc:377] Logged in from keytab as kudu/127.25.109.2@KRBTEST.COM (short username kudu)
I20260924 16:05:56.517491 11011 webserver.cc:460] Webserver started at http://127.25.109.2:33719/ using document root <none> and password file <none>
I20260924 16:05:56.518101 11011 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:56.518194 11011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:56.518421 11011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:56.520764 11011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-1/data/instance:
uuid: "0aa7771208754fa48e47374d18c87148"
format_stamp: "Formatted at 2026-09-24 16:05:56 on dist-test-slave-n106"
server_key: "96c580b94335a0e99397d85903fc4da3"
server_key_iv: "c7c0110bb8f7fc3ba445ce1e3e820c8c"
server_key_version: "encryptionkey@0"
I20260924 16:05:56.521359 11011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance:
uuid: "0aa7771208754fa48e47374d18c87148"
format_stamp: "Formatted at 2026-09-24 16:05:56 on dist-test-slave-n106"
server_key: "96c580b94335a0e99397d85903fc4da3"
server_key_iv: "c7c0110bb8f7fc3ba445ce1e3e820c8c"
server_key_version: "encryptionkey@0"
I20260924 16:05:56.525190 11011 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:05:56.527597 11027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:56.528904 11011 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:05:56.529055 11011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "0aa7771208754fa48e47374d18c87148"
format_stamp: "Formatted at 2026-09-24 16:05:56 on dist-test-slave-n106"
server_key: "96c580b94335a0e99397d85903fc4da3"
server_key_iv: "c7c0110bb8f7fc3ba445ce1e3e820c8c"
server_key_version: "encryptionkey@0"
I20260924 16:05:56.529148 11011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:56.546689 11011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:56.550158 11011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:56.550369 11011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:56.550997 11011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:05:56.551833 11011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:56.551879 11011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:56.551942 11011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:56.551985 11011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:56.560681 11011 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:37775
I20260924 16:05:56.560773 11141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:37775 every 8 connection(s)
I20260924 16:05:56.561621 11011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:05:56.570276 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 11011
I20260924 16:05:56.570415 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance
I20260924 16:05:56.570802 26036 external_mini_cluster.cc:1428] Setting key bcefaa93691f8ac3b9bdf27329d66789
Sep 24 16:05:56 dist-test-slave-n106 krb5kdc[10768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265956, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:05:56.575781 10853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:56.563427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:37969 (local address 127.25.109.62:33269)
0924 16:05:56.563563 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:56.563566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:56.564368 (+ 802us) server_negotiation.cc:407] Connection header received
0924 16:05:56.565713 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:56.565717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:56.565780 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:56.565885 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:56.567065 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.567595 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.568844 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.569064 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.569174 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:56.571908 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:56.571912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:56.571915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:56.571944 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:56.573579 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:56.574203 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:56.574205 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:56.574206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:56.574310 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:56.574704 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:56.574708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:56.574711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:56.574960 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:56.575067 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:56.575561 (+ 494us) server_negotiation.cc:299] Negotiation successful
0924 16:05:56.575663 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 16:05:56.576685 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:56.563763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33269 (local address 127.25.109.2:37969)
0924 16:05:56.564218 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:56.564261 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:56.565492 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:56.566057 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:56.566065 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:56.566421 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:56.566914 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:56.566928 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.567733 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.567742 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:56.568714 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:56.568724 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.569229 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.569233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:56.569391 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:56.569933 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:56.569949 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:56.571723 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:56.573697 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:56.573707 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:56.573725 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:56.574107 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:56.574434 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:56.574439 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:56.574442 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:56.574603 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:56.575065 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:56.575070 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:56.575312 (+ 242us) client_negotiation.cc:769] Sending connection context
0924 16:05:56.575472 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 16:05:56.575743 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":128,"threads_started":1}
I20260924 16:05:56.577911 11142 heartbeater.cc:347] Connected to a master server at 127.25.109.62:33269
I20260924 16:05:56.578284 11142 heartbeater.cc:464] Registering TS with master...
I20260924 16:05:56.578799 11142 heartbeater.cc:511] Master 127.25.109.62:33269 requested a full tablet report, sending...
I20260924 16:05:56.579893 10814 ts_manager.cc:194] Registered new tserver with Master: 0aa7771208754fa48e47374d18c87148 (127.25.109.2:37775)
I20260924 16:05:56.580547 10814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.2@KRBTEST.COM'} at 127.25.109.2:37969
WARNING: no policy specified for kudu/127.25.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
I20260924 16:05:56.599900 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:0
--local_ip_for_outbound_sockets=127.25.109.3
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:33269
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:33609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:56.711467 11147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:56.711733 11147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:56.711807 11147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:56.715744 11147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:56.715827 11147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:56.715915 11147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:05:56.721360 11147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:33609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:33269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:05:56.722412 11147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:56.723449 11147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:56.733033 11152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:56.733119 11147 server_base.cc:1034] running on GCE node
W20260924 16:05:56.733033 11155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:56.733292 11153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:56.733734 11147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:56.734365 11147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:56.735548 11147 hybrid_clock.cc:648] HybridClock initialized: now 1790265956735519 us; error 43 us; skew 500 ppm
Sep 24 16:05:56 dist-test-slave-n106 krb5kdc[10768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265956, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:56.738324 11147 init.cc:377] Logged in from keytab as kudu/127.25.109.3@KRBTEST.COM (short username kudu)
I20260924 16:05:56.739121 11147 webserver.cc:460] Webserver started at http://127.25.109.3:35743/ using document root <none> and password file <none>
I20260924 16:05:56.739643 11147 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:56.739702 11147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:56.739866 11147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:56.741911 11147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-2/data/instance:
uuid: "550007496dc04ed7b22fc46368ac6128"
format_stamp: "Formatted at 2026-09-24 16:05:56 on dist-test-slave-n106"
server_key: "2e0bd45fd22a1648ffc2f494f001ba68"
server_key_iv: "fafccc08db49077d896faa9ff2ffe898"
server_key_version: "encryptionkey@0"
I20260924 16:05:56.742429 11147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance:
uuid: "550007496dc04ed7b22fc46368ac6128"
format_stamp: "Formatted at 2026-09-24 16:05:56 on dist-test-slave-n106"
server_key: "2e0bd45fd22a1648ffc2f494f001ba68"
server_key_iv: "fafccc08db49077d896faa9ff2ffe898"
server_key_version: "encryptionkey@0"
I20260924 16:05:56.746179 11147 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:05:56.748540 11162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:56.749742 11147 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:05:56.749874 11147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "550007496dc04ed7b22fc46368ac6128"
format_stamp: "Formatted at 2026-09-24 16:05:56 on dist-test-slave-n106"
server_key: "2e0bd45fd22a1648ffc2f494f001ba68"
server_key_iv: "fafccc08db49077d896faa9ff2ffe898"
server_key_version: "encryptionkey@0"
I20260924 16:05:56.749981 11147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:56.777856 11147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:56.781345 11147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:56.781560 11147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:56.782193 11147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:05:56.783023 11147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:56.783068 11147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:56.783161 11147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:56.783201 11147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:56.791932 11147 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:41545
I20260924 16:05:56.791960 11277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:41545 every 8 connection(s)
I20260924 16:05:56.792919 11147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:05:56.796525 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 11147
I20260924 16:05:56.796681 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance
I20260924 16:05:56.796983 26036 external_mini_cluster.cc:1428] Setting key 0421fe75f8003c62d5e8debeda2b9042
Sep 24 16:05:56 dist-test-slave-n106 krb5kdc[10768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265956, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:05:56.809163 10853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:56.794698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:58959 (local address 127.25.109.62:33269)
0924 16:05:56.794837 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:56.794841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:56.795676 (+ 835us) server_negotiation.cc:407] Connection header received
0924 16:05:56.797220 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:56.797224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:56.797282 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:56.797382 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:56.798935 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.799600 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.801303 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.801691 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.801827 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:56.804788 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:56.804792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:56.804794 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:56.804821 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:56.806712 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:56.807347 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:56.807351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:56.807353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:56.807443 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:56.807828 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:56.807831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:56.807833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:56.808036 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:56.808149 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:56.808813 (+ 664us) server_negotiation.cc:299] Negotiation successful
0924 16:05:56.808959 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 16:05:56.810060 11280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:56.795016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:33269 (local address 127.25.109.3:58959)
0924 16:05:56.795493 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:56.795545 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:56.796970 (+ 1425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:56.797618 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:56.797625 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:56.798132 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:56.798790 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:56.798810 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.799717 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.799720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:56.801154 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:56.801169 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.801879 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.801883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:56.802068 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:56.802697 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:56.802713 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:56.804620 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:56.806851 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:56.806857 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:56.806872 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:56.807251 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:56.807587 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:56.807589 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:56.807591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:56.807720 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:56.808149 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:56.808157 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:56.808515 (+ 358us) client_negotiation.cc:769] Sending connection context
0924 16:05:56.808722 (+ 207us) client_negotiation.cc:240] Negotiation successful
0924 16:05:56.809017 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":130,"threads_started":1}
I20260924 16:05:56.811195 11278 heartbeater.cc:347] Connected to a master server at 127.25.109.62:33269
I20260924 16:05:56.811468 11278 heartbeater.cc:464] Registering TS with master...
I20260924 16:05:56.811926 11278 heartbeater.cc:511] Master 127.25.109.62:33269 requested a full tablet report, sending...
I20260924 16:05:56.812918 10814 ts_manager.cc:194] Registered new tserver with Master: 550007496dc04ed7b22fc46368ac6128 (127.25.109.3:41545)
I20260924 16:05:56.813486 10814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.3@KRBTEST.COM'} at 127.25.109.3:58959
I20260924 16:05:56.822860 26036 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:05:56.831756 10853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:56.824923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43718 (local address 127.25.109.62:33269)
0924 16:05:56.825061 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:56.825065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:56.825276 (+ 211us) server_negotiation.cc:407] Connection header received
0924 16:05:56.825377 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:56.825379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:56.825417 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:56.825479 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:56.826126 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.826611 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.827412 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.827617 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.827691 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:56.829076 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:56.829078 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:56.829080 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:56.829099 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:56.830291 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:56.830655 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:56.830657 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:56.830659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:56.830726 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:56.830946 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:56.830948 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:56.830949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:56.831154 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:56.831252 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:56.831455 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 16:05:56.831546 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 16:05:56.834532 10814 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:05:56.837522 10814 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:05:56.852138 11289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:56.846016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:41545 (local address 127.0.0.1:43630)
0924 16:05:56.846406 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:56.846461 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:56.846583 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:56.847987 (+ 1404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:56.847990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:56.848004 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:56.848267 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:56.848276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.849318 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.849325 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:56.850681 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:56.850693 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.851570 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.851573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:56.851699 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:56.851737 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 16:05:56.851847 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 16:05:56.851973 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":139,"thread_start_us":94,"threads_started":1}
I20260924 16:05:56.852191 11290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:56.846209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:37775 (local address 127.0.0.1:40508)
0924 16:05:56.846828 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:56.846840 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:56.846921 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:56.847836 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:56.847842 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:56.847865 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:56.848043 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:56.848050 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.849592 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.849595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:56.850726 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:56.850737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.851826 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.851833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:56.851914 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:56.851933 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 16:05:56.851988 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 16:05:56.852069 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":3,"thread_start_us":106,"threads_started":1}
I20260924 16:05:56.852512 11292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:56.846856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:36043 (local address 127.0.0.1:48724)
0924 16:05:56.847257 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:56.847269 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:56.847352 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:56.848051 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:56.848054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:56.848072 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:56.848188 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:56.848198 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.849638 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.849642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:56.850840 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:56.850850 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.852050 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.852053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:56.852169 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:56.852194 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 16:05:56.852272 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 16:05:56.852340 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":86,"threads_started":1}
I20260924 16:05:56.852540 11293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:56.846866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40508 (local address 127.25.109.2:37775)
0924 16:05:56.847308 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:56.847322 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:56.847346 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:05:56.847428 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:56.847436 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:56.847672 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:56.847826 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:56.848183 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.849448 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.850867 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.851717 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.851830 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:56.851968 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:56.852027 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 16:05:56.852134 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":125,"threads_started":1}
I20260924 16:05:56.852906 11291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:56.846180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43630 (local address 127.25.109.3:41545)
0924 16:05:56.847432 (+ 1252us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:56.847446 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:56.847472 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:05:56.847555 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:56.847561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:56.847836 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:56.848138 (+ 302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:56.848394 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.849185 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.850816 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.851443 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.851570 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:56.851730 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:56.852632 (+ 902us) server_negotiation.cc:299] Negotiation successful
0924 16:05:56.852743 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1152,"thread_start_us":147,"threads_started":1}
I20260924 16:05:56.853044 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:56.847014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48724 (local address 127.25.109.1:36043)
0924 16:05:56.847581 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:56.847597 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:56.847617 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:05:56.847689 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:56.847693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:56.847869 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:56.848054 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:56.848325 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.849448 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.850967 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.851717 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.851831 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:56.851968 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:56.852803 (+ 835us) server_negotiation.cc:299] Negotiation successful
0924 16:05:56.852908 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":116,"threads_started":1}
I20260924 16:05:56.855441 11076 tablet_service.cc:1467] Processing CreateTablet for tablet 87b40a17d9fb4c13b608f28a46f02e93 (DEFAULT_TABLE table=test-table [id=339e407284ca47d98bf60b1921e0febf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:05:56.855747 11210 tablet_service.cc:1467] Processing CreateTablet for tablet 87b40a17d9fb4c13b608f28a46f02e93 (DEFAULT_TABLE table=test-table [id=339e407284ca47d98bf60b1921e0febf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:05:56.855746 10940 tablet_service.cc:1467] Processing CreateTablet for tablet 87b40a17d9fb4c13b608f28a46f02e93 (DEFAULT_TABLE table=test-table [id=339e407284ca47d98bf60b1921e0febf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:05:56.856544 11076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87b40a17d9fb4c13b608f28a46f02e93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:56.856806 10940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87b40a17d9fb4c13b608f28a46f02e93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:56.856829 11210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87b40a17d9fb4c13b608f28a46f02e93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:56.862730 11295 tablet_bootstrap.cc:492] T 87b40a17d9fb4c13b608f28a46f02e93 P 0aa7771208754fa48e47374d18c87148: Bootstrap starting.
I20260924 16:05:56.864158 11297 tablet_bootstrap.cc:492] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128: Bootstrap starting.
I20260924 16:05:56.864562 11296 tablet_bootstrap.cc:492] T 87b40a17d9fb4c13b608f28a46f02e93 P 92b66c114e0e46b29f859f7478567499: Bootstrap starting.
I20260924 16:05:56.865231 11295 tablet_bootstrap.cc:654] T 87b40a17d9fb4c13b608f28a46f02e93 P 0aa7771208754fa48e47374d18c87148: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:56.866570 11295 log.cc:824] T 87b40a17d9fb4c13b608f28a46f02e93 P 0aa7771208754fa48e47374d18c87148: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:56.866801 11296 tablet_bootstrap.cc:654] T 87b40a17d9fb4c13b608f28a46f02e93 P 92b66c114e0e46b29f859f7478567499: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:56.867568 11296 log.cc:824] T 87b40a17d9fb4c13b608f28a46f02e93 P 92b66c114e0e46b29f859f7478567499: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:56.867635 11297 tablet_bootstrap.cc:654] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:56.868505 11297 log.cc:824] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:56.868729 11295 tablet_bootstrap.cc:492] T 87b40a17d9fb4c13b608f28a46f02e93 P 0aa7771208754fa48e47374d18c87148: No bootstrap required, opened a new log
I20260924 16:05:56.868923 11295 ts_tablet_manager.cc:1397] T 87b40a17d9fb4c13b608f28a46f02e93 P 0aa7771208754fa48e47374d18c87148: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260924 16:05:56.869638 11296 tablet_bootstrap.cc:492] T 87b40a17d9fb4c13b608f28a46f02e93 P 92b66c114e0e46b29f859f7478567499: No bootstrap required, opened a new log
I20260924 16:05:56.869834 11296 ts_tablet_manager.cc:1397] T 87b40a17d9fb4c13b608f28a46f02e93 P 92b66c114e0e46b29f859f7478567499: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 16:05:56.870257 11297 tablet_bootstrap.cc:492] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128: No bootstrap required, opened a new log
I20260924 16:05:56.870437 11297 ts_tablet_manager.cc:1397] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 16:05:56.873016 11295 raft_consensus.cc:348] T 87b40a17d9fb4c13b608f28a46f02e93 P 0aa7771208754fa48e47374d18c87148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "550007496dc04ed7b22fc46368ac6128" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41545 } } peers { permanent_uuid: "0aa7771208754fa48e47374d18c87148" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 37775 } } peers { permanent_uuid: "92b66c114e0e46b29f859f7478567499" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36043 } }
I20260924 16:05:56.873329 11295 raft_consensus.cc:374] T 87b40a17d9fb4c13b608f28a46f02e93 P 0aa7771208754fa48e47374d18c87148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:56.873426 11295 raft_consensus.cc:729] T 87b40a17d9fb4c13b608f28a46f02e93 P 0aa7771208754fa48e47374d18c87148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aa7771208754fa48e47374d18c87148, State: Initialized, Role: FOLLOWER
I20260924 16:05:56.873797 11297 raft_consensus.cc:348] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "550007496dc04ed7b22fc46368ac6128" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41545 } } peers { permanent_uuid: "0aa7771208754fa48e47374d18c87148" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 37775 } } peers { permanent_uuid: "92b66c114e0e46b29f859f7478567499" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36043 } }
I20260924 16:05:56.873903 11295 consensus_queue.cc:260] T 87b40a17d9fb4c13b608f28a46f02e93 P 0aa7771208754fa48e47374d18c87148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "550007496dc04ed7b22fc46368ac6128" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41545 } } peers { permanent_uuid: "0aa7771208754fa48e47374d18c87148" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 37775 } } peers { permanent_uuid: "92b66c114e0e46b29f859f7478567499" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36043 } }
I20260924 16:05:56.874022 11297 raft_consensus.cc:374] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:56.874100 11297 raft_consensus.cc:729] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 550007496dc04ed7b22fc46368ac6128, State: Initialized, Role: FOLLOWER
I20260924 16:05:56.874248 11296 raft_consensus.cc:348] T 87b40a17d9fb4c13b608f28a46f02e93 P 92b66c114e0e46b29f859f7478567499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "550007496dc04ed7b22fc46368ac6128" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41545 } } peers { permanent_uuid: "0aa7771208754fa48e47374d18c87148" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 37775 } } peers { permanent_uuid: "92b66c114e0e46b29f859f7478567499" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36043 } }
I20260924 16:05:56.874481 11296 raft_consensus.cc:374] T 87b40a17d9fb4c13b608f28a46f02e93 P 92b66c114e0e46b29f859f7478567499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:56.874534 11296 raft_consensus.cc:729] T 87b40a17d9fb4c13b608f28a46f02e93 P 92b66c114e0e46b29f859f7478567499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b66c114e0e46b29f859f7478567499, State: Initialized, Role: FOLLOWER
I20260924 16:05:56.874559 11297 consensus_queue.cc:260] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "550007496dc04ed7b22fc46368ac6128" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41545 } } peers { permanent_uuid: "0aa7771208754fa48e47374d18c87148" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 37775 } } peers { permanent_uuid: "92b66c114e0e46b29f859f7478567499" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36043 } }
I20260924 16:05:56.874926 11295 ts_tablet_manager.cc:1428] T 87b40a17d9fb4c13b608f28a46f02e93 P 0aa7771208754fa48e47374d18c87148: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 16:05:56.875214 11278 heartbeater.cc:503] Master 127.25.109.62:33269 was elected leader, sending a full tablet report...
I20260924 16:05:56.875326 11142 heartbeater.cc:503] Master 127.25.109.62:33269 was elected leader, sending a full tablet report...
I20260924 16:05:56.875509 11297 ts_tablet_manager.cc:1428] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 16:05:56.874974 11296 consensus_queue.cc:260] T 87b40a17d9fb4c13b608f28a46f02e93 P 92b66c114e0e46b29f859f7478567499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "550007496dc04ed7b22fc46368ac6128" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41545 } } peers { permanent_uuid: "0aa7771208754fa48e47374d18c87148" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 37775 } } peers { permanent_uuid: "92b66c114e0e46b29f859f7478567499" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36043 } }
I20260924 16:05:56.876189 11006 heartbeater.cc:503] Master 127.25.109.62:33269 was elected leader, sending a full tablet report...
I20260924 16:05:56.876511 11296 ts_tablet_manager.cc:1428] T 87b40a17d9fb4c13b608f28a46f02e93 P 92b66c114e0e46b29f859f7478567499: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260924 16:05:56.884701 11302 raft_consensus.cc:482] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:05:56.884828 11302 raft_consensus.cc:504] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "550007496dc04ed7b22fc46368ac6128" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41545 } } peers { permanent_uuid: "0aa7771208754fa48e47374d18c87148" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 37775 } } peers { permanent_uuid: "92b66c114e0e46b29f859f7478567499" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36043 } }
I20260924 16:05:56.885713 11302 leader_election.cc:288] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0aa7771208754fa48e47374d18c87148 (127.25.109.2:37775), 92b66c114e0e46b29f859f7478567499 (127.25.109.1:36043)
I20260924 16:05:56.889639 11280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:56.885918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:37775 (local address 127.25.109.3:33205)
0924 16:05:56.886016 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:56.886033 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:56.886111 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:56.886459 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:56.886461 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:56.886484 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:56.886569 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:56.886574 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.887535 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.887538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:56.888638 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:56.888649 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.889328 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.889331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:56.889410 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:56.889430 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 16:05:56.889488 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 16:05:56.889541 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 16:05:56.889739 11293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:56.886017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:33205 (local address 127.25.109.2:37775)
0924 16:05:56.886149 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:56.886156 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:56.886170 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:05:56.886214 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:56.886217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:56.886270 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:56.886344 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:56.886676 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.887434 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.888752 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.889220 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.889335 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:56.889385 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:56.889525 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 16:05:56.889591 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 16:05:56.890326 11096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87b40a17d9fb4c13b608f28a46f02e93" candidate_uuid: "550007496dc04ed7b22fc46368ac6128" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aa7771208754fa48e47374d18c87148" is_pre_election: true
I20260924 16:05:56.890393 11304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:56.885918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:36043 (local address 127.25.109.3:43163)
0924 16:05:56.886671 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:56.886687 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:56.886845 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:56.887092 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:56.887093 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:56.887109 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:56.887206 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:56.887210 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.887958 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.887962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:56.889006 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:56.889013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.889956 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.889960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:56.890060 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:56.890113 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 16:05:56.890190 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 16:05:56.890257 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":264,"thread_start_us":102,"threads_started":1}
I20260924 16:05:56.890388 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:56.886091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:43163 (local address 127.25.109.1:36043)
0924 16:05:56.886212 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:56.886217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:56.886776 (+ 559us) server_negotiation.cc:407] Connection header received
0924 16:05:56.886956 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:56.886959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:56.887001 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:56.887074 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:56.887302 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.887856 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.889122 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.889710 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.889827 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:56.889886 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:56.890221 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 16:05:56.890291 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 16:05:56.890619 11096 raft_consensus.cc:2454] T 87b40a17d9fb4c13b608f28a46f02e93 P 0aa7771208754fa48e47374d18c87148 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 550007496dc04ed7b22fc46368ac6128 in term 0.
I20260924 16:05:56.890964 10960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87b40a17d9fb4c13b608f28a46f02e93" candidate_uuid: "550007496dc04ed7b22fc46368ac6128" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92b66c114e0e46b29f859f7478567499" is_pre_election: true
I20260924 16:05:56.891130 11163 leader_election.cc:302] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aa7771208754fa48e47374d18c87148, 550007496dc04ed7b22fc46368ac6128; no voters:
I20260924 16:05:56.891166 10960 raft_consensus.cc:2454] T 87b40a17d9fb4c13b608f28a46f02e93 P 92b66c114e0e46b29f859f7478567499 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 550007496dc04ed7b22fc46368ac6128 in term 0.
I20260924 16:05:56.891494 11302 raft_consensus.cc:2781] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:05:56.891597 11302 raft_consensus.cc:482] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:05:56.891654 11302 raft_consensus.cc:3037] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:56.892730 11302 raft_consensus.cc:504] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "550007496dc04ed7b22fc46368ac6128" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41545 } } peers { permanent_uuid: "0aa7771208754fa48e47374d18c87148" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 37775 } } peers { permanent_uuid: "92b66c114e0e46b29f859f7478567499" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36043 } }
I20260924 16:05:56.893100 11302 leader_election.cc:288] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128 [CANDIDATE]: Term 1 election: Requested vote from peers 0aa7771208754fa48e47374d18c87148 (127.25.109.2:37775), 92b66c114e0e46b29f859f7478567499 (127.25.109.1:36043)
I20260924 16:05:56.893452 11096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87b40a17d9fb4c13b608f28a46f02e93" candidate_uuid: "550007496dc04ed7b22fc46368ac6128" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aa7771208754fa48e47374d18c87148"
I20260924 16:05:56.893488 10960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87b40a17d9fb4c13b608f28a46f02e93" candidate_uuid: "550007496dc04ed7b22fc46368ac6128" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92b66c114e0e46b29f859f7478567499"
I20260924 16:05:56.893553 11096 raft_consensus.cc:3037] T 87b40a17d9fb4c13b608f28a46f02e93 P 0aa7771208754fa48e47374d18c87148 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:56.893620 10960 raft_consensus.cc:3037] T 87b40a17d9fb4c13b608f28a46f02e93 P 92b66c114e0e46b29f859f7478567499 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:56.894620 10960 raft_consensus.cc:2454] T 87b40a17d9fb4c13b608f28a46f02e93 P 92b66c114e0e46b29f859f7478567499 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 550007496dc04ed7b22fc46368ac6128 in term 1.
I20260924 16:05:56.894644 11096 raft_consensus.cc:2454] T 87b40a17d9fb4c13b608f28a46f02e93 P 0aa7771208754fa48e47374d18c87148 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 550007496dc04ed7b22fc46368ac6128 in term 1.
I20260924 16:05:56.894927 11166 leader_election.cc:302] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 550007496dc04ed7b22fc46368ac6128, 92b66c114e0e46b29f859f7478567499; no voters:
I20260924 16:05:56.895123 11302 raft_consensus.cc:2781] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:05:56.895332 11302 raft_consensus.cc:686] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128 [term 1 LEADER]: Becoming Leader. State: Replica: 550007496dc04ed7b22fc46368ac6128, State: Running, Role: LEADER
I20260924 16:05:56.895635 11302 consensus_queue.cc:237] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "550007496dc04ed7b22fc46368ac6128" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41545 } } peers { permanent_uuid: "0aa7771208754fa48e47374d18c87148" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 37775 } } peers { permanent_uuid: "92b66c114e0e46b29f859f7478567499" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36043 } }
I20260924 16:05:56.898756 10814 catalog_manager.cc:5495] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128 reported cstate change: term changed from 0 to 1, leader changed from <none> to 550007496dc04ed7b22fc46368ac6128 (127.25.109.3). New cstate: current_term: 1 leader_uuid: "550007496dc04ed7b22fc46368ac6128" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "550007496dc04ed7b22fc46368ac6128" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0aa7771208754fa48e47374d18c87148" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 37775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92b66c114e0e46b29f859f7478567499" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36043 } health_report { overall_health: UNKNOWN } } }
I20260924 16:05:56.917250 11291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:56.914121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43632 (local address 127.25.109.3:41545)
0924 16:05:56.914246 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:56.914250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:56.914263 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:05:56.914406 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:56.914409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:56.914462 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:56.914547 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:56.914756 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.915240 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.916026 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.916231 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.916303 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:56.916561 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:56.916951 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:56.917039 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:56.917098 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 16:05:56.917145 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 16:05:56.923063 11096 raft_consensus.cc:1261] T 87b40a17d9fb4c13b608f28a46f02e93 P 0aa7771208754fa48e47374d18c87148 [term 1 FOLLOWER]: Refusing update from remote peer 550007496dc04ed7b22fc46368ac6128: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:05:56.923063 10960 raft_consensus.cc:1261] T 87b40a17d9fb4c13b608f28a46f02e93 P 92b66c114e0e46b29f859f7478567499 [term 1 FOLLOWER]: Refusing update from remote peer 550007496dc04ed7b22fc46368ac6128: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:05:56.923820 11302 consensus_queue.cc:1035] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aa7771208754fa48e47374d18c87148" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 37775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:05:56.924032 11305 consensus_queue.cc:1035] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b66c114e0e46b29f859f7478567499" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 36043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:05:56.932067 11309 mvcc.cc:205] Tried to move back new op lower bound from 7332929359549755392 to 7332929359447191552. Current Snapshot: MvccSnapshot[applied={T|T < 7332929359549755392}]
I20260924 16:05:56.932536 11311 mvcc.cc:205] Tried to move back new op lower bound from 7332929359549755392 to 7332929359447191552. Current Snapshot: MvccSnapshot[applied={T|T < 7332929359549755392 or (T in {7332929359549755392})}]
I20260924 16:05:56.939854 10813 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:05:56.940027 10813 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:05:56.942441 10813 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e80cc00a493d4fe8b698dd383d6cead2: Sending DeleteTablet for 3 replicas of tablet 87b40a17d9fb4c13b608f28a46f02e93
I20260924 16:05:56.943171 11210 tablet_service.cc:1514] Processing DeleteTablet for tablet 87b40a17d9fb4c13b608f28a46f02e93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:05:56 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:43630
I20260924 16:05:56.943188 11076 tablet_service.cc:1514] Processing DeleteTablet for tablet 87b40a17d9fb4c13b608f28a46f02e93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:05:56 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:40508
I20260924 16:05:56.943348 10940 tablet_service.cc:1514] Processing DeleteTablet for tablet 87b40a17d9fb4c13b608f28a46f02e93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:05:56 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:48724
I20260924 16:05:56.943787 11318 tablet_replica.cc:330] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128: stopping tablet replica
I20260924 16:05:56.943792 11320 tablet_replica.cc:330] T 87b40a17d9fb4c13b608f28a46f02e93 P 92b66c114e0e46b29f859f7478567499: stopping tablet replica
I20260924 16:05:56.944099 11320 raft_consensus.cc:2229] T 87b40a17d9fb4c13b608f28a46f02e93 P 92b66c114e0e46b29f859f7478567499 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:05:56.944118 11318 raft_consensus.cc:2229] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:05:56.944511 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 10876
I20260924 16:05:56.944501 11320 raft_consensus.cc:2258] T 87b40a17d9fb4c13b608f28a46f02e93 P 92b66c114e0e46b29f859f7478567499 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:05:56.946036 11318 raft_consensus.cc:2258] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:05:56.947580 11318 ts_tablet_manager.cc:1905] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:05:56.951025 11318 ts_tablet_manager.cc:1918] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:05:56.951125 11318 log.cc:1196] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265708917108-26036-0/minicluster-data/ts-2/wal/wals/87b40a17d9fb4c13b608f28a46f02e93
I20260924 16:05:56.951457 11318 ts_tablet_manager.cc:1939] T 87b40a17d9fb4c13b608f28a46f02e93 P 550007496dc04ed7b22fc46368ac6128: Deleting consensus metadata
W20260924 16:05:56.952564 10801 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.109.1:36043 (error 108)
I20260924 16:05:56.952776 10799 catalog_manager.cc:4841] TS 550007496dc04ed7b22fc46368ac6128 (127.25.109.3:41545): tablet 87b40a17d9fb4c13b608f28a46f02e93 (table test-table [id=339e407284ca47d98bf60b1921e0febf]) successfully deleted
I20260924 16:05:56.953559 11292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:56.953260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:36043 (local address 127.0.0.1:48726)
0924 16:05:56.953422 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:56.953490 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.1:36043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260924 16:05:56.953711 10801 catalog_manager.cc:4584] TS 92b66c114e0e46b29f859f7478567499 (127.25.109.1:36043): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 87b40a17d9fb4c13b608f28a46f02e93: Network error: Client connection negotiation failed: client connection to 127.25.109.1:36043: connect: Connection refused (error 111)
I20260924 16:05:56.956624 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 11011
I20260924 16:05:56.956694 11319 tablet_replica.cc:330] T 87b40a17d9fb4c13b608f28a46f02e93 P 0aa7771208754fa48e47374d18c87148: stopping tablet replica
I20260924 16:05:56.962244 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 11147
I20260924 16:05:56.963439 11292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:56.962392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:37775 (local address 127.0.0.1:40514)
0924 16:05:56.963254 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:56.963321 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.2:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":757}
I20260924 16:05:56.968362 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 10781
2026-09-24T16:05:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2268 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T16:05:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:05:56Z Disabled control of system clock
I20260924 16:05:56.998533 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:34127
--webserver_interface=127.25.109.62
--webserver_port=0
--builtin_ntp_servers=127.25.109.20:43277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:34127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 16:05:57.107645 11325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:57.107870 11325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:57.107903 11325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:57.111833 11325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:05:57.111884 11325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:57.111900 11325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:57.111917 11325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:05:57.111931 11325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:05:57.116638 11325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:43277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:34127
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:34127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:05:57.117594 11325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:57.118894 11325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:57.127236 11332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:57.127275 11331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:57.127410 11325 server_base.cc:1034] running on GCE node
W20260924 16:05:57.127295 11334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:57.128165 11325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:57.129246 11325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:57.130412 11325 hybrid_clock.cc:648] HybridClock initialized: now 1790265957130393 us; error 29 us; skew 500 ppm
I20260924 16:05:57.132131 11325 webserver.cc:460] Webserver started at http://127.25.109.62:45883/ using document root <none> and password file <none>
I20260924 16:05:57.132730 11325 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:57.132794 11325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:57.132953 11325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:57.134971 11325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "272a62268a1c488da0c0d5b0b5e26331"
format_stamp: "Formatted at 2026-09-24 16:05:57 on dist-test-slave-n106"
server_key: "9d200a729a6450b6f89d8310f1fc25e7"
server_key_iv: "90bfd694dbfe987858bf2a001e903b29"
server_key_version: "encryptionkey@0"
I20260924 16:05:57.135489 11325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "272a62268a1c488da0c0d5b0b5e26331"
format_stamp: "Formatted at 2026-09-24 16:05:57 on dist-test-slave-n106"
server_key: "9d200a729a6450b6f89d8310f1fc25e7"
server_key_iv: "90bfd694dbfe987858bf2a001e903b29"
server_key_version: "encryptionkey@0"
I20260924 16:05:57.139150 11325 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 16:05:57.141561 11340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:57.142781 11325 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:05:57.142894 11325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "272a62268a1c488da0c0d5b0b5e26331"
format_stamp: "Formatted at 2026-09-24 16:05:57 on dist-test-slave-n106"
server_key: "9d200a729a6450b6f89d8310f1fc25e7"
server_key_iv: "90bfd694dbfe987858bf2a001e903b29"
server_key_version: "encryptionkey@0"
I20260924 16:05:57.142984 11325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:57.160091 11325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:57.160873 11325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:57.161037 11325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:57.167863 11325 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:34127
I20260924 16:05:57.167889 11392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:34127 every 8 connection(s)
I20260924 16:05:57.168908 11325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:05:57.172056 11393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:57.173921 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 11325
I20260924 16:05:57.174057 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:05:57.174377 26036 external_mini_cluster.cc:1428] Setting key b70a2058b04e7a9cd2b7a93adbd60fcd
I20260924 16:05:57.178434 11393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 272a62268a1c488da0c0d5b0b5e26331: Bootstrap starting.
I20260924 16:05:57.181125 11393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 272a62268a1c488da0c0d5b0b5e26331: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:57.182063 11393 log.cc:824] T 00000000000000000000000000000000 P 272a62268a1c488da0c0d5b0b5e26331: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:57.184273 11393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 272a62268a1c488da0c0d5b0b5e26331: No bootstrap required, opened a new log
I20260924 16:05:57.184621 11396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:57.175897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52130 (local address 127.25.109.62:34127)
0924 16:05:57.176593 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:57.176610 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:57.176648 (+ 38us) server_negotiation.cc:407] Connection header received
0924 16:05:57.177466 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:57.177487 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:57.177916 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:57.178246 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:57.178677 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.179769 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.180951 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.181482 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.181615 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:57.182596 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:57.182603 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:57.182620 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:05:57.183127 (+ 507us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:57.183187 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:05:57.183208 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:57.183287 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:57.183466 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 16:05:57.183774 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":145,"threads_started":1}
I20260924 16:05:57.187393 11393 raft_consensus.cc:348] T 00000000000000000000000000000000 P 272a62268a1c488da0c0d5b0b5e26331 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "272a62268a1c488da0c0d5b0b5e26331" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 34127 } }
I20260924 16:05:57.187659 11393 raft_consensus.cc:374] T 00000000000000000000000000000000 P 272a62268a1c488da0c0d5b0b5e26331 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:57.187748 11393 raft_consensus.cc:729] T 00000000000000000000000000000000 P 272a62268a1c488da0c0d5b0b5e26331 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 272a62268a1c488da0c0d5b0b5e26331, State: Initialized, Role: FOLLOWER
I20260924 16:05:57.188235 11393 consensus_queue.cc:260] T 00000000000000000000000000000000 P 272a62268a1c488da0c0d5b0b5e26331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "272a62268a1c488da0c0d5b0b5e26331" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 34127 } }
I20260924 16:05:57.188366 11393 raft_consensus.cc:388] T 00000000000000000000000000000000 P 272a62268a1c488da0c0d5b0b5e26331 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:05:57.188472 11393 raft_consensus.cc:482] T 00000000000000000000000000000000 P 272a62268a1c488da0c0d5b0b5e26331 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:05:57.188575 11393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 272a62268a1c488da0c0d5b0b5e26331 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:57.189548 11393 raft_consensus.cc:504] T 00000000000000000000000000000000 P 272a62268a1c488da0c0d5b0b5e26331 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "272a62268a1c488da0c0d5b0b5e26331" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 34127 } }
I20260924 16:05:57.189934 11393 leader_election.cc:302] T 00000000000000000000000000000000 P 272a62268a1c488da0c0d5b0b5e26331 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 272a62268a1c488da0c0d5b0b5e26331; no voters:
I20260924 16:05:57.190337 11393 leader_election.cc:288] T 00000000000000000000000000000000 P 272a62268a1c488da0c0d5b0b5e26331 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:05:57.190412 11398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 272a62268a1c488da0c0d5b0b5e26331 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:05:57.190637 11398 raft_consensus.cc:686] T 00000000000000000000000000000000 P 272a62268a1c488da0c0d5b0b5e26331 [term 1 LEADER]: Becoming Leader. State: Replica: 272a62268a1c488da0c0d5b0b5e26331, State: Running, Role: LEADER
I20260924 16:05:57.191005 11398 consensus_queue.cc:237] T 00000000000000000000000000000000 P 272a62268a1c488da0c0d5b0b5e26331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "272a62268a1c488da0c0d5b0b5e26331" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 34127 } }
I20260924 16:05:57.191730 11393 sys_catalog.cc:564] T 00000000000000000000000000000000 P 272a62268a1c488da0c0d5b0b5e26331 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:05:57.192961 11399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 272a62268a1c488da0c0d5b0b5e26331 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "272a62268a1c488da0c0d5b0b5e26331" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "272a62268a1c488da0c0d5b0b5e26331" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 34127 } } }
I20260924 16:05:57.192940 11400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 272a62268a1c488da0c0d5b0b5e26331 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 272a62268a1c488da0c0d5b0b5e26331. Latest consensus state: current_term: 1 leader_uuid: "272a62268a1c488da0c0d5b0b5e26331" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "272a62268a1c488da0c0d5b0b5e26331" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 34127 } } }
I20260924 16:05:57.193161 11399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 272a62268a1c488da0c0d5b0b5e26331 [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:57.193274 11400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 272a62268a1c488da0c0d5b0b5e26331 [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:57.193730 11407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:05:57.196630 11407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:05:57.203354 11407 catalog_manager.cc:1347] Generated new cluster ID: 304f959bf5fd4fbda2980a6770bb1d3f
I20260924 16:05:57.203449 11407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:05:57.218592 11407 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:05:57.219883 11407 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:05:57.228588 11407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 272a62268a1c488da0c0d5b0b5e26331: Generated new TSK 0
I20260924 16:05:57.229528 11407 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:05:57.237648 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:0
--local_ip_for_outbound_sockets=127.25.109.1
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:34127
--builtin_ntp_servers=127.25.109.20:43277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 16:05:57.352579 11418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:57.352862 11418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:57.352902 11418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:57.357146 11418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:57.357203 11418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:57.357299 11418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:05:57.362177 11418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:43277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:34127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:05:57.363330 11418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:57.364949 11418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:57.375174 11427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:57.375166 11424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:57.375291 11418 server_base.cc:1034] running on GCE node
W20260924 16:05:57.375174 11425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:57.375887 11418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:57.376705 11418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:57.377864 11418 hybrid_clock.cc:648] HybridClock initialized: now 1790265957377826 us; error 52 us; skew 500 ppm
I20260924 16:05:57.379894 11418 webserver.cc:460] Webserver started at http://127.25.109.1:37909/ using document root <none> and password file <none>
I20260924 16:05:57.380610 11418 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:57.380681 11418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:57.380944 11418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:57.383191 11418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-0/data/instance:
uuid: "7f5acd038f6b46978171af7a0593d126"
format_stamp: "Formatted at 2026-09-24 16:05:57 on dist-test-slave-n106"
server_key: "0c4e03b5566e0da26ec0a3b62ce6b847"
server_key_iv: "d227944195593d12a12d1e577c65b33a"
server_key_version: "encryptionkey@0"
I20260924 16:05:57.383836 11418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance:
uuid: "7f5acd038f6b46978171af7a0593d126"
format_stamp: "Formatted at 2026-09-24 16:05:57 on dist-test-slave-n106"
server_key: "0c4e03b5566e0da26ec0a3b62ce6b847"
server_key_iv: "d227944195593d12a12d1e577c65b33a"
server_key_version: "encryptionkey@0"
I20260924 16:05:57.387709 11418 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260924 16:05:57.390322 11433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:57.391695 11418 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 16:05:57.391829 11418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "7f5acd038f6b46978171af7a0593d126"
format_stamp: "Formatted at 2026-09-24 16:05:57 on dist-test-slave-n106"
server_key: "0c4e03b5566e0da26ec0a3b62ce6b847"
server_key_iv: "d227944195593d12a12d1e577c65b33a"
server_key_version: "encryptionkey@0"
I20260924 16:05:57.391947 11418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:57.401736 11418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:57.402434 11418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:57.402639 11418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:57.403229 11418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:05:57.404212 11418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:57.404284 11418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:57.404349 11418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:57.404400 11418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:57.413136 11418 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:43995
I20260924 16:05:57.413200 11546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:43995 every 8 connection(s)
I20260924 16:05:57.414090 11418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:05:57.414803 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 11418
I20260924 16:05:57.414935 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance
I20260924 16:05:57.415324 26036 external_mini_cluster.cc:1428] Setting key 2664299f7c44278844ea899c06cc926d
I20260924 16:05:57.417382 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:0
--local_ip_for_outbound_sockets=127.25.109.2
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:34127
--builtin_ntp_servers=127.25.109.20:43277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 16:05:57.424798 11396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:57.416133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:56337 (local address 127.25.109.62:34127)
0924 16:05:57.416283 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:57.416289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:57.418227 (+ 1938us) server_negotiation.cc:407] Connection header received
0924 16:05:57.419697 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:57.419700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:57.419758 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:57.419860 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:57.420957 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.421548 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.422715 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.422944 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.423058 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:57.424251 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:57.424255 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:57.424258 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:05:57.424271 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:57.424301 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:05:57.424309 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:57.424390 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:57.424581 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 16:05:57.424649 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:05:57.425675 11549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:57.416687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:34127 (local address 127.25.109.1:56337)
0924 16:05:57.418086 (+ 1399us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:57.418126 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:57.419462 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:57.420051 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:57.420059 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:57.420186 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:57.420809 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:57.420823 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.421683 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.421686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:57.422572 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:57.422590 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.423102 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.423107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:57.423303 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:57.423971 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:05:57.423973 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:57.424017 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:05:57.424088 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:57.424393 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:57.424428 (+ 35us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:57.424435 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 16:05:57.424585 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 16:05:57.424875 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"thread_start_us":139,"threads_started":1}
I20260924 16:05:57.427146 11547 heartbeater.cc:347] Connected to a master server at 127.25.109.62:34127
I20260924 16:05:57.427390 11547 heartbeater.cc:464] Registering TS with master...
I20260924 16:05:57.427923 11547 heartbeater.cc:511] Master 127.25.109.62:34127 requested a full tablet report, sending...
I20260924 16:05:57.429651 11357 ts_manager.cc:194] Registered new tserver with Master: 7f5acd038f6b46978171af7a0593d126 (127.25.109.1:43995)
I20260924 16:05:57.431272 11357 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.1:56337
W20260924 16:05:57.532120 11550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:57.532382 11550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:57.532455 11550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:57.536494 11550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:57.536540 11550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:57.536593 11550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:05:57.541301 11550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:43277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:34127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:05:57.542449 11550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:57.543957 11550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:57.553915 11560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:57.553931 11556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:57.553930 11557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:57.554159 11550 server_base.cc:1034] running on GCE node
I20260924 16:05:57.554593 11550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:57.555243 11550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:57.556452 11550 hybrid_clock.cc:648] HybridClock initialized: now 1790265957556383 us; error 34 us; skew 500 ppm
I20260924 16:05:57.558336 11550 webserver.cc:460] Webserver started at http://127.25.109.2:39551/ using document root <none> and password file <none>
I20260924 16:05:57.558935 11550 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:57.558996 11550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:57.559218 11550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:57.561385 11550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-1/data/instance:
uuid: "e22f39dc8fb64651a523cc58f873feb4"
format_stamp: "Formatted at 2026-09-24 16:05:57 on dist-test-slave-n106"
server_key: "2a0655647cdcf7f8f742fbf414745fb8"
server_key_iv: "76ba5db4937c6dc3760d89f6386041d6"
server_key_version: "encryptionkey@0"
I20260924 16:05:57.561990 11550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance:
uuid: "e22f39dc8fb64651a523cc58f873feb4"
format_stamp: "Formatted at 2026-09-24 16:05:57 on dist-test-slave-n106"
server_key: "2a0655647cdcf7f8f742fbf414745fb8"
server_key_iv: "76ba5db4937c6dc3760d89f6386041d6"
server_key_version: "encryptionkey@0"
I20260924 16:05:57.565820 11550 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:05:57.568208 11566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:57.569481 11550 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:05:57.569641 11550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "e22f39dc8fb64651a523cc58f873feb4"
format_stamp: "Formatted at 2026-09-24 16:05:57 on dist-test-slave-n106"
server_key: "2a0655647cdcf7f8f742fbf414745fb8"
server_key_iv: "76ba5db4937c6dc3760d89f6386041d6"
server_key_version: "encryptionkey@0"
I20260924 16:05:57.569762 11550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:57.581321 11550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:57.581982 11550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:57.582181 11550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:57.582760 11550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:05:57.583793 11550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:57.583863 11550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:57.583941 11550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:57.583984 11550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:57.593058 11550 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:46199
I20260924 16:05:57.593091 11681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:46199 every 8 connection(s)
I20260924 16:05:57.594153 11550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:05:57.603863 11396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:57.596096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:33697 (local address 127.25.109.62:34127)
0924 16:05:57.596252 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:57.596256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:57.597415 (+ 1159us) server_negotiation.cc:407] Connection header received
0924 16:05:57.598900 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:57.598904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:57.598986 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:57.599082 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:57.600024 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.600647 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.601839 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.602081 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.602200 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:57.603317 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:57.603321 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:57.603326 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:05:57.603343 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:57.603362 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:05:57.603370 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:57.603457 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:57.603640 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 16:05:57.603706 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 16:05:57.604049 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 11550
I20260924 16:05:57.604177 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance
I20260924 16:05:57.604569 26036 external_mini_cluster.cc:1428] Setting key 002c7f4e56f6ddd2dd68d1de3e5e7592
I20260924 16:05:57.604955 11684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:57.596592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:34127 (local address 127.25.109.2:33697)
0924 16:05:57.597262 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:57.597303 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:57.598664 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:57.599279 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:57.599286 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:57.599376 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:57.599868 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:57.599880 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.600787 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.600791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:57.601703 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:57.601713 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.602247 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.602251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:57.602424 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:57.603043 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:05:57.603044 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:57.603085 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:05:57.603156 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:57.603485 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:57.603500 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:57.603504 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 16:05:57.603638 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 16:05:57.603926 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":119,"threads_started":1}
I20260924 16:05:57.606194 11682 heartbeater.cc:347] Connected to a master server at 127.25.109.62:34127
I20260924 16:05:57.606449 11682 heartbeater.cc:464] Registering TS with master...
I20260924 16:05:57.606909 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:0
--local_ip_for_outbound_sockets=127.25.109.3
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:34127
--builtin_ntp_servers=127.25.109.20:43277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 16:05:57.607132 11682 heartbeater.cc:511] Master 127.25.109.62:34127 requested a full tablet report, sending...
I20260924 16:05:57.608502 11357 ts_manager.cc:194] Registered new tserver with Master: e22f39dc8fb64651a523cc58f873feb4 (127.25.109.2:46199)
I20260924 16:05:57.609375 11357 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.2:33697
W20260924 16:05:57.722659 11685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:57.722978 11685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:57.723050 11685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:57.727423 11685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:57.727523 11685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:57.727617 11685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:05:57.732632 11685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:43277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:34127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:05:57.733906 11685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:57.735328 11685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:57.745332 11694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:57.745342 11692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:57.745589 11685 server_base.cc:1034] running on GCE node
W20260924 16:05:57.745355 11690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:57.746147 11685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:57.746908 11685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:57.748140 11685 hybrid_clock.cc:648] HybridClock initialized: now 1790265957748096 us; error 43 us; skew 500 ppm
I20260924 16:05:57.750267 11685 webserver.cc:460] Webserver started at http://127.25.109.3:44329/ using document root <none> and password file <none>
I20260924 16:05:57.750872 11685 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:57.750936 11685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:57.751104 11685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:57.753332 11685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-2/data/instance:
uuid: "e28b2fe8a5b7488fb27f6c1b19c97c6f"
format_stamp: "Formatted at 2026-09-24 16:05:57 on dist-test-slave-n106"
server_key: "d7f6abb7059ad007f933e304bedc3818"
server_key_iv: "680251661671c1d10a977210e6b1957a"
server_key_version: "encryptionkey@0"
I20260924 16:05:57.753934 11685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance:
uuid: "e28b2fe8a5b7488fb27f6c1b19c97c6f"
format_stamp: "Formatted at 2026-09-24 16:05:57 on dist-test-slave-n106"
server_key: "d7f6abb7059ad007f933e304bedc3818"
server_key_iv: "680251661671c1d10a977210e6b1957a"
server_key_version: "encryptionkey@0"
I20260924 16:05:57.757799 11685 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:05:57.760564 11701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:57.761863 11685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:05:57.761996 11685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "e28b2fe8a5b7488fb27f6c1b19c97c6f"
format_stamp: "Formatted at 2026-09-24 16:05:57 on dist-test-slave-n106"
server_key: "d7f6abb7059ad007f933e304bedc3818"
server_key_iv: "680251661671c1d10a977210e6b1957a"
server_key_version: "encryptionkey@0"
I20260924 16:05:57.762151 11685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:57.824092 11685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:57.824906 11685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:57.825136 11685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:57.825814 11685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:05:57.826848 11685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:57.826895 11685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:57.826936 11685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:57.826993 11685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:57.836342 11685 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:38737
I20260924 16:05:57.836369 11814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:38737 every 8 connection(s)
I20260924 16:05:57.837378 11685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:05:57.845498 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 11685
I20260924 16:05:57.845676 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance
I20260924 16:05:57.846060 26036 external_mini_cluster.cc:1428] Setting key fddc819d2fb0fa2dd319c92e94f61232
I20260924 16:05:57.847662 11396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:57.839310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:55757 (local address 127.25.109.62:34127)
0924 16:05:57.839460 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:57.839465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:57.840625 (+ 1160us) server_negotiation.cc:407] Connection header received
0924 16:05:57.842001 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:57.842004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:57.842051 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:57.842133 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:57.843055 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.843611 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.844916 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.845146 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.845231 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:57.846966 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:57.847002 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:57.847008 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:05:57.847037 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:57.847066 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:05:57.847079 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:57.847163 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:57.847422 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 16:05:57.847496 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 16:05:57.848744 11817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:57.839748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:34127 (local address 127.25.109.3:55757)
0924 16:05:57.840463 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:57.840522 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:57.841804 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:57.842302 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:57.842309 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:57.842396 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:57.842918 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:57.842931 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.843758 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.843761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:57.844793 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:57.844805 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.845412 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.845417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:57.845650 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:57.846504 (+ 854us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:05:57.846506 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:57.846584 (+ 78us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:05:57.846677 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:57.847185 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:57.847195 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:57.847202 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 16:05:57.847356 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 16:05:57.847738 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":116,"threads_started":1}
I20260924 16:05:57.849864 11815 heartbeater.cc:347] Connected to a master server at 127.25.109.62:34127
I20260924 16:05:57.850132 11815 heartbeater.cc:464] Registering TS with master...
I20260924 16:05:57.850692 11815 heartbeater.cc:511] Master 127.25.109.62:34127 requested a full tablet report, sending...
I20260924 16:05:57.851740 11357 ts_manager.cc:194] Registered new tserver with Master: e28b2fe8a5b7488fb27f6c1b19c97c6f (127.25.109.3:38737)
I20260924 16:05:57.852336 11357 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.3:55757
I20260924 16:05:57.860301 26036 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:05:57.865967 11396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:57.862300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52138 (local address 127.25.109.62:34127)
0924 16:05:57.862402 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:57.862406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:57.862649 (+ 243us) server_negotiation.cc:407] Connection header received
0924 16:05:57.862725 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:57.862727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:57.862770 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:57.862835 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:57.863034 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.863547 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.864400 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.864690 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.864781 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:57.865479 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:57.865482 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:57.865484 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:05:57.865496 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:57.865516 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:05:57.865524 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:57.865602 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:57.865689 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 16:05:57.865742 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 16:05:57.868690 11357 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:05:57.871759 11357 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:05:57.887133 11830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:57.881534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40048 (local address 127.25.109.3:38737)
0924 16:05:57.882071 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:57.882084 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:57.882108 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:05:57.882185 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:57.882190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:57.882378 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:57.882556 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:57.882795 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.883973 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.885592 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.886340 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.886464 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:57.886667 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:57.886817 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 16:05:57.886939 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":159,"threads_started":1}
I20260924 16:05:57.887149 11829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:57.881526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:43995 (local address 127.0.0.1:51430)
0924 16:05:57.882402 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:57.882415 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:57.882497 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:57.883201 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:57.883204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:57.883267 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:57.883362 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:57.883369 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.884718 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.884721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:57.885779 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:57.885787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.886590 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.886593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:57.886662 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:57.886679 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 16:05:57.886757 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 16:05:57.886910 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"mutex_wait_us":7,"thread_start_us":88,"threads_started":1}
I20260924 16:05:57.887147 11828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:57.881125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:38737 (local address 127.0.0.1:40048)
0924 16:05:57.881589 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:57.881603 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:57.881684 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:57.882532 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:57.882535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:57.882555 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:57.882668 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:57.882675 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.884096 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.884100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:57.885477 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:57.885488 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.886480 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.886486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:57.886618 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:57.886666 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 16:05:57.886788 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 16:05:57.886951 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":125,"thread_start_us":94,"threads_started":1}
I20260924 16:05:57.887506 11827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:57.880818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51370 (local address 127.25.109.2:46199)
0924 16:05:57.881284 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:57.881299 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:57.881601 (+ 302us) server_negotiation.cc:407] Connection header received
0924 16:05:57.881695 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:57.881703 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:57.881926 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:57.882079 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:57.882618 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.883742 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.885504 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.886340 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.886464 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:57.886667 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:57.887217 (+ 550us) server_negotiation.cc:299] Negotiation successful
0924 16:05:57.887351 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":111,"threads_started":1}
I20260924 16:05:57.888392 11831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:57.881973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51430 (local address 127.25.109.1:43995)
0924 16:05:57.882761 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:57.882773 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:57.882795 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:05:57.882866 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:57.882870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:57.883036 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:57.883189 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:57.883509 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.884590 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.885898 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.886462 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.887796 (+ 1334us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:57.888055 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:57.888110 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 16:05:57.888217 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":149,"threads_started":1}
I20260924 16:05:57.888535 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:57.880776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:46199 (local address 127.0.0.1:51370)
0924 16:05:57.881226 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:57.881279 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:57.881431 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:57.882278 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:57.882287 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:57.882317 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:57.882461 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:57.882469 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.883858 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.883861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:57.885365 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:57.885375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.886861 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.886864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:57.886938 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:57.886958 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 16:05:57.887025 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 16:05:57.887099 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":202,"thread_start_us":82,"threads_started":1}
I20260924 16:05:57.891098 11749 tablet_service.cc:1467] Processing CreateTablet for tablet 26779fa3bf9746009d738d3eaa72b4a4 (DEFAULT_TABLE table=test-table [id=6ee70182986f411c9edf0dd3a344bd22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:05:57.891084 11481 tablet_service.cc:1467] Processing CreateTablet for tablet 26779fa3bf9746009d738d3eaa72b4a4 (DEFAULT_TABLE table=test-table [id=6ee70182986f411c9edf0dd3a344bd22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:05:57.891290 11616 tablet_service.cc:1467] Processing CreateTablet for tablet 26779fa3bf9746009d738d3eaa72b4a4 (DEFAULT_TABLE table=test-table [id=6ee70182986f411c9edf0dd3a344bd22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:05:57.892313 11481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26779fa3bf9746009d738d3eaa72b4a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:57.892313 11749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26779fa3bf9746009d738d3eaa72b4a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:57.892338 11616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26779fa3bf9746009d738d3eaa72b4a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:57.899029 11832 tablet_bootstrap.cc:492] T 26779fa3bf9746009d738d3eaa72b4a4 P e22f39dc8fb64651a523cc58f873feb4: Bootstrap starting.
I20260924 16:05:57.900699 11834 tablet_bootstrap.cc:492] T 26779fa3bf9746009d738d3eaa72b4a4 P e28b2fe8a5b7488fb27f6c1b19c97c6f: Bootstrap starting.
I20260924 16:05:57.900674 11833 tablet_bootstrap.cc:492] T 26779fa3bf9746009d738d3eaa72b4a4 P 7f5acd038f6b46978171af7a0593d126: Bootstrap starting.
I20260924 16:05:57.901716 11832 tablet_bootstrap.cc:654] T 26779fa3bf9746009d738d3eaa72b4a4 P e22f39dc8fb64651a523cc58f873feb4: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:57.903083 11834 tablet_bootstrap.cc:654] T 26779fa3bf9746009d738d3eaa72b4a4 P e28b2fe8a5b7488fb27f6c1b19c97c6f: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:57.903083 11833 tablet_bootstrap.cc:654] T 26779fa3bf9746009d738d3eaa72b4a4 P 7f5acd038f6b46978171af7a0593d126: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:57.903107 11832 log.cc:824] T 26779fa3bf9746009d738d3eaa72b4a4 P e22f39dc8fb64651a523cc58f873feb4: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:57.904026 11834 log.cc:824] T 26779fa3bf9746009d738d3eaa72b4a4 P e28b2fe8a5b7488fb27f6c1b19c97c6f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:57.904026 11833 log.cc:824] T 26779fa3bf9746009d738d3eaa72b4a4 P 7f5acd038f6b46978171af7a0593d126: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:57.905344 11832 tablet_bootstrap.cc:492] T 26779fa3bf9746009d738d3eaa72b4a4 P e22f39dc8fb64651a523cc58f873feb4: No bootstrap required, opened a new log
I20260924 16:05:57.905567 11832 ts_tablet_manager.cc:1397] T 26779fa3bf9746009d738d3eaa72b4a4 P e22f39dc8fb64651a523cc58f873feb4: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260924 16:05:57.906222 11833 tablet_bootstrap.cc:492] T 26779fa3bf9746009d738d3eaa72b4a4 P 7f5acd038f6b46978171af7a0593d126: No bootstrap required, opened a new log
I20260924 16:05:57.906347 11834 tablet_bootstrap.cc:492] T 26779fa3bf9746009d738d3eaa72b4a4 P e28b2fe8a5b7488fb27f6c1b19c97c6f: No bootstrap required, opened a new log
I20260924 16:05:57.906425 11833 ts_tablet_manager.cc:1397] T 26779fa3bf9746009d738d3eaa72b4a4 P 7f5acd038f6b46978171af7a0593d126: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 16:05:57.906543 11834 ts_tablet_manager.cc:1397] T 26779fa3bf9746009d738d3eaa72b4a4 P e28b2fe8a5b7488fb27f6c1b19c97c6f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 16:05:57.909054 11832 raft_consensus.cc:348] T 26779fa3bf9746009d738d3eaa72b4a4 P e22f39dc8fb64651a523cc58f873feb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22f39dc8fb64651a523cc58f873feb4" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46199 } } peers { permanent_uuid: "7f5acd038f6b46978171af7a0593d126" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 43995 } } peers { permanent_uuid: "e28b2fe8a5b7488fb27f6c1b19c97c6f" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 38737 } }
I20260924 16:05:57.909350 11832 raft_consensus.cc:374] T 26779fa3bf9746009d738d3eaa72b4a4 P e22f39dc8fb64651a523cc58f873feb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:57.909425 11832 raft_consensus.cc:729] T 26779fa3bf9746009d738d3eaa72b4a4 P e22f39dc8fb64651a523cc58f873feb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e22f39dc8fb64651a523cc58f873feb4, State: Initialized, Role: FOLLOWER
I20260924 16:05:57.910207 11832 consensus_queue.cc:260] T 26779fa3bf9746009d738d3eaa72b4a4 P e22f39dc8fb64651a523cc58f873feb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22f39dc8fb64651a523cc58f873feb4" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46199 } } peers { permanent_uuid: "7f5acd038f6b46978171af7a0593d126" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 43995 } } peers { permanent_uuid: "e28b2fe8a5b7488fb27f6c1b19c97c6f" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 38737 } }
I20260924 16:05:57.910923 11682 heartbeater.cc:503] Master 127.25.109.62:34127 was elected leader, sending a full tablet report...
I20260924 16:05:57.911008 11834 raft_consensus.cc:348] T 26779fa3bf9746009d738d3eaa72b4a4 P e28b2fe8a5b7488fb27f6c1b19c97c6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22f39dc8fb64651a523cc58f873feb4" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46199 } } peers { permanent_uuid: "7f5acd038f6b46978171af7a0593d126" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 43995 } } peers { permanent_uuid: "e28b2fe8a5b7488fb27f6c1b19c97c6f" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 38737 } }
I20260924 16:05:57.911268 11834 raft_consensus.cc:374] T 26779fa3bf9746009d738d3eaa72b4a4 P e28b2fe8a5b7488fb27f6c1b19c97c6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:57.911322 11834 raft_consensus.cc:729] T 26779fa3bf9746009d738d3eaa72b4a4 P e28b2fe8a5b7488fb27f6c1b19c97c6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e28b2fe8a5b7488fb27f6c1b19c97c6f, State: Initialized, Role: FOLLOWER
I20260924 16:05:57.911340 11832 ts_tablet_manager.cc:1428] T 26779fa3bf9746009d738d3eaa72b4a4 P e22f39dc8fb64651a523cc58f873feb4: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260924 16:05:57.911748 11834 consensus_queue.cc:260] T 26779fa3bf9746009d738d3eaa72b4a4 P e28b2fe8a5b7488fb27f6c1b19c97c6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22f39dc8fb64651a523cc58f873feb4" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46199 } } peers { permanent_uuid: "7f5acd038f6b46978171af7a0593d126" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 43995 } } peers { permanent_uuid: "e28b2fe8a5b7488fb27f6c1b19c97c6f" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 38737 } }
I20260924 16:05:57.911823 11833 raft_consensus.cc:348] T 26779fa3bf9746009d738d3eaa72b4a4 P 7f5acd038f6b46978171af7a0593d126 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22f39dc8fb64651a523cc58f873feb4" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46199 } } peers { permanent_uuid: "7f5acd038f6b46978171af7a0593d126" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 43995 } } peers { permanent_uuid: "e28b2fe8a5b7488fb27f6c1b19c97c6f" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 38737 } }
I20260924 16:05:57.912108 11833 raft_consensus.cc:374] T 26779fa3bf9746009d738d3eaa72b4a4 P 7f5acd038f6b46978171af7a0593d126 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:57.912160 11833 raft_consensus.cc:729] T 26779fa3bf9746009d738d3eaa72b4a4 P 7f5acd038f6b46978171af7a0593d126 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f5acd038f6b46978171af7a0593d126, State: Initialized, Role: FOLLOWER
I20260924 16:05:57.912590 11833 consensus_queue.cc:260] T 26779fa3bf9746009d738d3eaa72b4a4 P 7f5acd038f6b46978171af7a0593d126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22f39dc8fb64651a523cc58f873feb4" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46199 } } peers { permanent_uuid: "7f5acd038f6b46978171af7a0593d126" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 43995 } } peers { permanent_uuid: "e28b2fe8a5b7488fb27f6c1b19c97c6f" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 38737 } }
I20260924 16:05:57.912736 11834 ts_tablet_manager.cc:1428] T 26779fa3bf9746009d738d3eaa72b4a4 P e28b2fe8a5b7488fb27f6c1b19c97c6f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 16:05:57.913069 11815 heartbeater.cc:503] Master 127.25.109.62:34127 was elected leader, sending a full tablet report...
I20260924 16:05:57.913241 11547 heartbeater.cc:503] Master 127.25.109.62:34127 was elected leader, sending a full tablet report...
I20260924 16:05:57.913501 11833 ts_tablet_manager.cc:1428] T 26779fa3bf9746009d738d3eaa72b4a4 P 7f5acd038f6b46978171af7a0593d126: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260924 16:05:57.915254 11548 tablet.cc:2369] T 26779fa3bf9746009d738d3eaa72b4a4 P 7f5acd038f6b46978171af7a0593d126: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:05:57.977035 11841 raft_consensus.cc:482] T 26779fa3bf9746009d738d3eaa72b4a4 P 7f5acd038f6b46978171af7a0593d126 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:05:57.977273 11841 raft_consensus.cc:504] T 26779fa3bf9746009d738d3eaa72b4a4 P 7f5acd038f6b46978171af7a0593d126 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22f39dc8fb64651a523cc58f873feb4" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46199 } } peers { permanent_uuid: "7f5acd038f6b46978171af7a0593d126" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 43995 } } peers { permanent_uuid: "e28b2fe8a5b7488fb27f6c1b19c97c6f" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 38737 } }
I20260924 16:05:57.978449 11841 leader_election.cc:288] T 26779fa3bf9746009d738d3eaa72b4a4 P 7f5acd038f6b46978171af7a0593d126 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e22f39dc8fb64651a523cc58f873feb4 (127.25.109.2:46199), e28b2fe8a5b7488fb27f6c1b19c97c6f (127.25.109.3:38737)
I20260924 16:05:57.982738 11830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:57.978846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:39119 (local address 127.25.109.3:38737)
0924 16:05:57.978958 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:57.978964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:57.979174 (+ 210us) server_negotiation.cc:407] Connection header received
0924 16:05:57.979293 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:57.979295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:57.979334 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:57.979424 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:57.979763 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.980255 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.981694 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.982134 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.982268 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:57.982331 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:57.982561 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 16:05:57.982634 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 16:05:57.982738 11827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:57.978768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:56481 (local address 127.25.109.2:46199)
0924 16:05:57.978916 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:57.978922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:57.979035 (+ 113us) server_negotiation.cc:407] Connection header received
0924 16:05:57.979249 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:57.979256 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:57.979314 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:57.979380 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:57.979748 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.980259 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.981661 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.982106 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.982229 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:57.982291 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:57.982506 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 16:05:57.982593 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:05:57.982764 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:57.978635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:46199 (local address 127.25.109.1:56481)
0924 16:05:57.978946 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:57.978963 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:57.979082 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:57.979419 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:57.979422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:57.979437 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:57.979623 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:57.979629 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.980391 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.980394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:57.981531 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:57.981541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.982225 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.982229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:57.982361 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:57.982385 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 16:05:57.982455 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 16:05:57.982545 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":132,"spinlock_wait_cycles":38144,"thread_start_us":76,"threads_started":1}
I20260924 16:05:57.982769 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:57.978762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:38737 (local address 127.25.109.1:39119)
0924 16:05:57.979069 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:57.979082 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:57.979202 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:57.979454 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:57.979457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:57.979504 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:57.979625 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:57.979631 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.980366 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.980372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:57.981529 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:57.981540 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.982284 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.982288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:57.982411 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:57.982428 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 16:05:57.982527 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 16:05:57.982608 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":31,"spinlock_wait_cycles":4224,"thread_start_us":110,"threads_started":1}
I20260924 16:05:57.983268 11840 raft_consensus.cc:482] T 26779fa3bf9746009d738d3eaa72b4a4 P e28b2fe8a5b7488fb27f6c1b19c97c6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:05:57.983479 11636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26779fa3bf9746009d738d3eaa72b4a4" candidate_uuid: "7f5acd038f6b46978171af7a0593d126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e22f39dc8fb64651a523cc58f873feb4" is_pre_election: true
I20260924 16:05:57.983489 11840 raft_consensus.cc:504] T 26779fa3bf9746009d738d3eaa72b4a4 P e28b2fe8a5b7488fb27f6c1b19c97c6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22f39dc8fb64651a523cc58f873feb4" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46199 } } peers { permanent_uuid: "7f5acd038f6b46978171af7a0593d126" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 43995 } } peers { permanent_uuid: "e28b2fe8a5b7488fb27f6c1b19c97c6f" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 38737 } }
I20260924 16:05:57.983796 11769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26779fa3bf9746009d738d3eaa72b4a4" candidate_uuid: "7f5acd038f6b46978171af7a0593d126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e28b2fe8a5b7488fb27f6c1b19c97c6f" is_pre_election: true
I20260924 16:05:57.983857 11636 raft_consensus.cc:2454] T 26779fa3bf9746009d738d3eaa72b4a4 P e22f39dc8fb64651a523cc58f873feb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f5acd038f6b46978171af7a0593d126 in term 0.
I20260924 16:05:57.983997 11769 raft_consensus.cc:2454] T 26779fa3bf9746009d738d3eaa72b4a4 P e28b2fe8a5b7488fb27f6c1b19c97c6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f5acd038f6b46978171af7a0593d126 in term 0.
I20260924 16:05:57.984447 11435 leader_election.cc:302] T 26779fa3bf9746009d738d3eaa72b4a4 P 7f5acd038f6b46978171af7a0593d126 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f5acd038f6b46978171af7a0593d126, e22f39dc8fb64651a523cc58f873feb4; no voters:
I20260924 16:05:57.984853 11840 leader_election.cc:288] T 26779fa3bf9746009d738d3eaa72b4a4 P e28b2fe8a5b7488fb27f6c1b19c97c6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e22f39dc8fb64651a523cc58f873feb4 (127.25.109.2:46199), 7f5acd038f6b46978171af7a0593d126 (127.25.109.1:43995)
I20260924 16:05:57.984997 11841 raft_consensus.cc:2781] T 26779fa3bf9746009d738d3eaa72b4a4 P 7f5acd038f6b46978171af7a0593d126 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:05:57.985121 11841 raft_consensus.cc:482] T 26779fa3bf9746009d738d3eaa72b4a4 P 7f5acd038f6b46978171af7a0593d126 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:05:57.985183 11841 raft_consensus.cc:3037] T 26779fa3bf9746009d738d3eaa72b4a4 P 7f5acd038f6b46978171af7a0593d126 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:57.986543 11841 raft_consensus.cc:504] T 26779fa3bf9746009d738d3eaa72b4a4 P 7f5acd038f6b46978171af7a0593d126 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22f39dc8fb64651a523cc58f873feb4" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46199 } } peers { permanent_uuid: "7f5acd038f6b46978171af7a0593d126" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 43995 } } peers { permanent_uuid: "e28b2fe8a5b7488fb27f6c1b19c97c6f" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 38737 } }
I20260924 16:05:57.987293 11769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26779fa3bf9746009d738d3eaa72b4a4" candidate_uuid: "7f5acd038f6b46978171af7a0593d126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e28b2fe8a5b7488fb27f6c1b19c97c6f"
I20260924 16:05:57.987432 11769 raft_consensus.cc:3037] T 26779fa3bf9746009d738d3eaa72b4a4 P e28b2fe8a5b7488fb27f6c1b19c97c6f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:57.988595 11769 raft_consensus.cc:2454] T 26779fa3bf9746009d738d3eaa72b4a4 P e28b2fe8a5b7488fb27f6c1b19c97c6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f5acd038f6b46978171af7a0593d126 in term 1.
I20260924 16:05:57.988590 11636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26779fa3bf9746009d738d3eaa72b4a4" candidate_uuid: "7f5acd038f6b46978171af7a0593d126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e22f39dc8fb64651a523cc58f873feb4"
I20260924 16:05:57.988732 11636 raft_consensus.cc:3037] T 26779fa3bf9746009d738d3eaa72b4a4 P e22f39dc8fb64651a523cc58f873feb4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:57.988819 11841 leader_election.cc:288] T 26779fa3bf9746009d738d3eaa72b4a4 P 7f5acd038f6b46978171af7a0593d126 [CANDIDATE]: Term 1 election: Requested vote from peers e22f39dc8fb64651a523cc58f873feb4 (127.25.109.2:46199), e28b2fe8a5b7488fb27f6c1b19c97c6f (127.25.109.3:38737)
I20260924 16:05:57.988997 11437 leader_election.cc:302] T 26779fa3bf9746009d738d3eaa72b4a4 P 7f5acd038f6b46978171af7a0593d126 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f5acd038f6b46978171af7a0593d126, e28b2fe8a5b7488fb27f6c1b19c97c6f; no voters:
I20260924 16:05:57.989166 11841 raft_consensus.cc:2781] T 26779fa3bf9746009d738d3eaa72b4a4 P 7f5acd038f6b46978171af7a0593d126 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:05:57.989429 11841 raft_consensus.cc:686] T 26779fa3bf9746009d738d3eaa72b4a4 P 7f5acd038f6b46978171af7a0593d126 [term 1 LEADER]: Becoming Leader. State: Replica: 7f5acd038f6b46978171af7a0593d126, State: Running, Role: LEADER
I20260924 16:05:57.989863 11841 consensus_queue.cc:237] T 26779fa3bf9746009d738d3eaa72b4a4 P 7f5acd038f6b46978171af7a0593d126 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22f39dc8fb64651a523cc58f873feb4" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46199 } } peers { permanent_uuid: "7f5acd038f6b46978171af7a0593d126" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 43995 } } peers { permanent_uuid: "e28b2fe8a5b7488fb27f6c1b19c97c6f" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 38737 } }
I20260924 16:05:57.990068 11831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:57.985812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:48701 (local address 127.25.109.1:43995)
0924 16:05:57.985918 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:57.985923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:57.985935 (+ 12us) server_negotiation.cc:407] Connection header received
0924 16:05:57.985981 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:57.985983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:57.986029 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:57.986107 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:57.986720 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.987392 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.988816 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.989362 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.989461 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:57.989513 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:57.989876 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 16:05:57.989958 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 16:05:57.990115 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:57.985318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:43995 (local address 127.25.109.3:48701)
0924 16:05:57.985639 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:57.985652 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:57.985744 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:57.986416 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:57.986421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:57.986443 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:57.986582 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:57.986588 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.987523 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.987527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:57.988669 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:57.988679 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.989627 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.989631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:57.989738 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:57.989762 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 16:05:57.989845 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 16:05:57.989922 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":148,"threads_started":1}
I20260924 16:05:57.990171 11636 raft_consensus.cc:2454] T 26779fa3bf9746009d738d3eaa72b4a4 P e22f39dc8fb64651a523cc58f873feb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f5acd038f6b46978171af7a0593d126 in term 1.
I20260924 16:05:57.990986 11501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26779fa3bf9746009d738d3eaa72b4a4" candidate_uuid: "e28b2fe8a5b7488fb27f6c1b19c97c6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f5acd038f6b46978171af7a0593d126" is_pre_election: true
I20260924 16:05:57.991915 11817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:57.985043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:46199 (local address 127.25.109.3:50597)
0924 16:05:57.985164 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:57.985181 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:57.985265 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:57.986145 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:57.986152 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:57.986173 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:57.986279 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:57.986286 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.987113 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.987116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:57.988354 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:57.988366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.991467 (+ 3101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.991471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:57.991604 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:57.991645 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 16:05:57.991723 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 16:05:57.991799 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 16:05:57.992471 11827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:57.985756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:50597 (local address 127.25.109.2:46199)
0924 16:05:57.985944 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:57.985949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:57.985960 (+ 11us) server_negotiation.cc:407] Connection header received
0924 16:05:57.985995 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:57.985998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:57.986036 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:57.986118 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:57.986378 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.986973 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.990787 (+ 3814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.991336 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.991463 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:57.991530 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:57.992225 (+ 695us) server_negotiation.cc:299] Negotiation successful
0924 16:05:57.992328 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260924 16:05:57.992926 11636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26779fa3bf9746009d738d3eaa72b4a4" candidate_uuid: "e28b2fe8a5b7488fb27f6c1b19c97c6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e22f39dc8fb64651a523cc58f873feb4" is_pre_election: true
I20260924 16:05:57.993140 11636 raft_consensus.cc:2379] T 26779fa3bf9746009d738d3eaa72b4a4 P e22f39dc8fb64651a523cc58f873feb4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e28b2fe8a5b7488fb27f6c1b19c97c6f in current term 1: Already voted for candidate 7f5acd038f6b46978171af7a0593d126 in this term.
I20260924 16:05:57.993646 11703 leader_election.cc:302] T 26779fa3bf9746009d738d3eaa72b4a4 P e28b2fe8a5b7488fb27f6c1b19c97c6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e28b2fe8a5b7488fb27f6c1b19c97c6f; no voters: 7f5acd038f6b46978171af7a0593d126, e22f39dc8fb64651a523cc58f873feb4
I20260924 16:05:57.993969 11840 raft_consensus.cc:2726] T 26779fa3bf9746009d738d3eaa72b4a4 P e28b2fe8a5b7488fb27f6c1b19c97c6f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 16:05:57.993876 11356 catalog_manager.cc:5495] T 26779fa3bf9746009d738d3eaa72b4a4 P 7f5acd038f6b46978171af7a0593d126 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f5acd038f6b46978171af7a0593d126 (127.25.109.1). New cstate: current_term: 1 leader_uuid: "7f5acd038f6b46978171af7a0593d126" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22f39dc8fb64651a523cc58f873feb4" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f5acd038f6b46978171af7a0593d126" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 43995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e28b2fe8a5b7488fb27f6c1b19c97c6f" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 38737 } health_report { overall_health: UNKNOWN } } }
I20260924 16:05:58.008721 11831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:58.005386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51436 (local address 127.25.109.1:43995)
0924 16:05:58.005523 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:58.005527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:58.005540 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:05:58.005641 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:58.005643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:58.005698 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:58.005782 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:58.006003 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:58.006493 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:58.007306 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:58.007524 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:58.007652 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:58.007905 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:58.008338 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:58.008491 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:58.008549 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 16:05:58.008596 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 16:05:58.014725 11636 raft_consensus.cc:1261] T 26779fa3bf9746009d738d3eaa72b4a4 P e22f39dc8fb64651a523cc58f873feb4 [term 1 FOLLOWER]: Refusing update from remote peer 7f5acd038f6b46978171af7a0593d126: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:05:58.014768 11769 raft_consensus.cc:1261] T 26779fa3bf9746009d738d3eaa72b4a4 P e28b2fe8a5b7488fb27f6c1b19c97c6f [term 1 FOLLOWER]: Refusing update from remote peer 7f5acd038f6b46978171af7a0593d126: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:05:58.015416 11841 consensus_queue.cc:1035] T 26779fa3bf9746009d738d3eaa72b4a4 P 7f5acd038f6b46978171af7a0593d126 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e28b2fe8a5b7488fb27f6c1b19c97c6f" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:05:58.015622 11845 consensus_queue.cc:1035] T 26779fa3bf9746009d738d3eaa72b4a4 P 7f5acd038f6b46978171af7a0593d126 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e22f39dc8fb64651a523cc58f873feb4" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 46199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:05:58.022568 11848 mvcc.cc:205] Tried to move back new op lower bound from 7332929364021080064 to 7332929363930091520. Current Snapshot: MvccSnapshot[applied={T|T < 7332929364021080064}]
I20260924 16:05:58.025888 11849 mvcc.cc:205] Tried to move back new op lower bound from 7332929364021080064 to 7332929363930091520. Current Snapshot: MvccSnapshot[applied={T|T < 7332929364021080064}]
I20260924 16:05:58.032881 11357 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:05:58.033054 11357 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:05:58.035778 11357 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 272a62268a1c488da0c0d5b0b5e26331: Sending DeleteTablet for 3 replicas of tablet 26779fa3bf9746009d738d3eaa72b4a4
I20260924 16:05:58.036727 11616 tablet_service.cc:1514] Processing DeleteTablet for tablet 26779fa3bf9746009d738d3eaa72b4a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:05:58 UTC) from {username='slave'} at 127.0.0.1:51370
I20260924 16:05:58.036728 11749 tablet_service.cc:1514] Processing DeleteTablet for tablet 26779fa3bf9746009d738d3eaa72b4a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:05:58 UTC) from {username='slave'} at 127.0.0.1:40048
I20260924 16:05:58.036798 11481 tablet_service.cc:1514] Processing DeleteTablet for tablet 26779fa3bf9746009d738d3eaa72b4a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:05:58 UTC) from {username='slave'} at 127.0.0.1:51430
I20260924 16:05:58.037310 11858 tablet_replica.cc:330] T 26779fa3bf9746009d738d3eaa72b4a4 P e22f39dc8fb64651a523cc58f873feb4: stopping tablet replica
I20260924 16:05:58.037321 11857 tablet_replica.cc:330] T 26779fa3bf9746009d738d3eaa72b4a4 P e28b2fe8a5b7488fb27f6c1b19c97c6f: stopping tablet replica
I20260924 16:05:58.037672 11857 raft_consensus.cc:2229] T 26779fa3bf9746009d738d3eaa72b4a4 P e28b2fe8a5b7488fb27f6c1b19c97c6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:05:58.037698 11858 raft_consensus.cc:2229] T 26779fa3bf9746009d738d3eaa72b4a4 P e22f39dc8fb64651a523cc58f873feb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:05:58.038022 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 11418
I20260924 16:05:58.038049 11857 raft_consensus.cc:2258] T 26779fa3bf9746009d738d3eaa72b4a4 P e28b2fe8a5b7488fb27f6c1b19c97c6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:05:58.039132 11858 raft_consensus.cc:2258] T 26779fa3bf9746009d738d3eaa72b4a4 P e22f39dc8fb64651a523cc58f873feb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:05:58.041256 11857 ts_tablet_manager.cc:1905] T 26779fa3bf9746009d738d3eaa72b4a4 P e28b2fe8a5b7488fb27f6c1b19c97c6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:05:58.041256 11858 ts_tablet_manager.cc:1905] T 26779fa3bf9746009d738d3eaa72b4a4 P e22f39dc8fb64651a523cc58f873feb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:05:58.044181 11858 ts_tablet_manager.cc:1918] T 26779fa3bf9746009d738d3eaa72b4a4 P e22f39dc8fb64651a523cc58f873feb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:05:58.044181 11857 ts_tablet_manager.cc:1918] T 26779fa3bf9746009d738d3eaa72b4a4 P e28b2fe8a5b7488fb27f6c1b19c97c6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:05:58.044296 11857 log.cc:1196] T 26779fa3bf9746009d738d3eaa72b4a4 P e28b2fe8a5b7488fb27f6c1b19c97c6f: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-2/wal/wals/26779fa3bf9746009d738d3eaa72b4a4
I20260924 16:05:58.044296 11858 log.cc:1196] T 26779fa3bf9746009d738d3eaa72b4a4 P e22f39dc8fb64651a523cc58f873feb4: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265708917108-26036-0/minicluster-data/ts-1/wal/wals/26779fa3bf9746009d738d3eaa72b4a4
I20260924 16:05:58.044664 11858 ts_tablet_manager.cc:1939] T 26779fa3bf9746009d738d3eaa72b4a4 P e22f39dc8fb64651a523cc58f873feb4: Deleting consensus metadata
I20260924 16:05:58.044664 11857 ts_tablet_manager.cc:1939] T 26779fa3bf9746009d738d3eaa72b4a4 P e28b2fe8a5b7488fb27f6c1b19c97c6f: Deleting consensus metadata
I20260924 16:05:58.045601 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 11550
W20260924 16:05:58.045818 11342 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.109.1:43995 (error 108)
I20260924 16:05:58.047675 11344 catalog_manager.cc:4841] TS e28b2fe8a5b7488fb27f6c1b19c97c6f (127.25.109.3:38737): tablet 26779fa3bf9746009d738d3eaa72b4a4 (table test-table [id=6ee70182986f411c9edf0dd3a344bd22]) successfully deleted
I20260924 16:05:58.048396 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:58.048092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:43995 (local address 127.0.0.1:51442)
0924 16:05:58.048245 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:58.048314 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.1:43995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260924 16:05:58.048691 11342 catalog_manager.cc:4584] TS 7f5acd038f6b46978171af7a0593d126 (127.25.109.1:43995): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 26779fa3bf9746009d738d3eaa72b4a4: Network error: Client connection negotiation failed: client connection to 127.25.109.1:43995: connect: Connection refused (error 111)
I20260924 16:05:58.052343 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 11685
I20260924 16:05:58.054188 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:58.052624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:46199 (local address 127.0.0.1:51384)
0924 16:05:58.054035 (+ 1411us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:58.054107 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.2:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1306}
I20260924 16:05:58.058560 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 11325
2026-09-24T16:05:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1089 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:05:58 dist-test-slave-n106 krb5kdc[11863](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:05:58 dist-test-slave-n106 krb5kdc[11863](info): set up 2 sockets
Sep 24 16:05:58 dist-test-slave-n106 krb5kdc[11863](info): commencing operation
krb5kdc: starting...
W20260924 16:05:59.117856 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:05:59 dist-test-slave-n106 krb5kdc[11863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:05:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:05:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
I20260924 16:05:59.242995 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:38619
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:38619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:59.354946 11876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:59.355284 11876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:59.355361 11876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:59.359422 11876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:05:59.359516 11876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:59.359556 11876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:59.359596 11876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:05:59.359630 11876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:05:59.365011 11876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:38619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:38619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:05:59.366184 11876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:59.367264 11876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:59.375926 11882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:59.375922 11884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:59.375975 11881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:59.376370 11876 server_base.cc:1034] running on GCE node
I20260924 16:05:59.376971 11876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:59.378111 11876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:59.379309 11876 hybrid_clock.cc:648] HybridClock initialized: now 1790265959379278 us; error 50 us; skew 500 ppm
Sep 24 16:05:59 dist-test-slave-n106 krb5kdc[11863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265959, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:59.382179 11876 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:05:59.383104 11876 webserver.cc:460] Webserver started at http://127.25.109.62:36681/ using document root <none> and password file <none>
I20260924 16:05:59.383708 11876 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:59.383769 11876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:59.384014 11876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:59.386277 11876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "98688153dc664d3ba3a073503b285b20"
format_stamp: "Formatted at 2026-09-24 16:05:59 on dist-test-slave-n106"
server_key: "89282527c84613682461422dfb24b3f8"
server_key_iv: "62e1a0b6b61389d92df916d860511316"
server_key_version: "encryptionkey@0"
I20260924 16:05:59.386848 11876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "98688153dc664d3ba3a073503b285b20"
format_stamp: "Formatted at 2026-09-24 16:05:59 on dist-test-slave-n106"
server_key: "89282527c84613682461422dfb24b3f8"
server_key_iv: "62e1a0b6b61389d92df916d860511316"
server_key_version: "encryptionkey@0"
I20260924 16:05:59.390582 11876 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 16:05:59.393128 11891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:59.394404 11876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 16:05:59.394559 11876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "98688153dc664d3ba3a073503b285b20"
format_stamp: "Formatted at 2026-09-24 16:05:59 on dist-test-slave-n106"
server_key: "89282527c84613682461422dfb24b3f8"
server_key_iv: "62e1a0b6b61389d92df916d860511316"
server_key_version: "encryptionkey@0"
I20260924 16:05:59.394690 11876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:59.426370 11876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:59.429787 11876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:59.430047 11876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:59.437603 11876 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:38619
I20260924 16:05:59.437626 11944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:38619 every 8 connection(s)
I20260924 16:05:59.438659 11876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:05:59.439651 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 11876
I20260924 16:05:59.439793 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:05:59.440086 26036 external_mini_cluster.cc:1428] Setting key a3020f0de26c39420e4b6807d10e99d2
I20260924 16:05:59.442019 11945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:59.449353 11945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98688153dc664d3ba3a073503b285b20: Bootstrap starting.
Sep 24 16:05:59 dist-test-slave-n106 krb5kdc[11863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:05:59.452554 11945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98688153dc664d3ba3a073503b285b20: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:59.453576 11945 log.cc:824] T 00000000000000000000000000000000 P 98688153dc664d3ba3a073503b285b20: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:59.456019 11948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:59.441559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39544 (local address 127.25.109.62:38619)
0924 16:05:59.441995 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:59.442014 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:59.442059 (+ 45us) server_negotiation.cc:407] Connection header received
0924 16:05:59.442761 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:59.442779 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:59.443133 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:59.443444 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:59.444456 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:59.445503 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:59.446714 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:59.447195 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:59.447313 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:59.450285 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:59.450291 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:59.450309 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:59.450346 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:59.452529 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:59.453045 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:59.453051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:59.453058 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:59.453210 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:59.453485 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:59.453489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:59.453490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:59.454090 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:59.454256 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:59.454672 (+ 416us) server_negotiation.cc:299] Negotiation successful
0924 16:05:59.454996 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":148,"threads_started":1}
I20260924 16:05:59.456188 11945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98688153dc664d3ba3a073503b285b20: No bootstrap required, opened a new log
I20260924 16:05:59.460173 11945 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98688153dc664d3ba3a073503b285b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98688153dc664d3ba3a073503b285b20" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 38619 } }
I20260924 16:05:59.460462 11945 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98688153dc664d3ba3a073503b285b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:59.460546 11945 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98688153dc664d3ba3a073503b285b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98688153dc664d3ba3a073503b285b20, State: Initialized, Role: FOLLOWER
I20260924 16:05:59.460986 11945 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98688153dc664d3ba3a073503b285b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98688153dc664d3ba3a073503b285b20" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 38619 } }
I20260924 16:05:59.461118 11945 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98688153dc664d3ba3a073503b285b20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:05:59.461165 11945 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98688153dc664d3ba3a073503b285b20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:05:59.461239 11945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98688153dc664d3ba3a073503b285b20 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:59.462164 11945 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98688153dc664d3ba3a073503b285b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98688153dc664d3ba3a073503b285b20" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 38619 } }
I20260924 16:05:59.462472 11945 leader_election.cc:302] T 00000000000000000000000000000000 P 98688153dc664d3ba3a073503b285b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98688153dc664d3ba3a073503b285b20; no voters:
I20260924 16:05:59.462802 11945 leader_election.cc:288] T 00000000000000000000000000000000 P 98688153dc664d3ba3a073503b285b20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:05:59.462950 11950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98688153dc664d3ba3a073503b285b20 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:05:59.463204 11950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98688153dc664d3ba3a073503b285b20 [term 1 LEADER]: Becoming Leader. State: Replica: 98688153dc664d3ba3a073503b285b20, State: Running, Role: LEADER
I20260924 16:05:59.463539 11950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98688153dc664d3ba3a073503b285b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98688153dc664d3ba3a073503b285b20" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 38619 } }
I20260924 16:05:59.464172 11945 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98688153dc664d3ba3a073503b285b20 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:05:59.465435 11951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98688153dc664d3ba3a073503b285b20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98688153dc664d3ba3a073503b285b20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98688153dc664d3ba3a073503b285b20" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 38619 } } }
I20260924 16:05:59.465554 11951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98688153dc664d3ba3a073503b285b20 [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:59.465848 11952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98688153dc664d3ba3a073503b285b20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98688153dc664d3ba3a073503b285b20. Latest consensus state: current_term: 1 leader_uuid: "98688153dc664d3ba3a073503b285b20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98688153dc664d3ba3a073503b285b20" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 38619 } } }
I20260924 16:05:59.465938 11952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98688153dc664d3ba3a073503b285b20 [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:59.465962 11960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:05:59.469252 11960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:05:59.475679 11960 catalog_manager.cc:1347] Generated new cluster ID: e2c80c433b8c4ce68cafb670d62e65a8
I20260924 16:05:59.475773 11960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:05:59.484961 11960 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:05:59.486060 11960 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:05:59.492332 11960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98688153dc664d3ba3a073503b285b20: Generated new TSK 0
I20260924 16:05:59.493178 11960 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
I20260924 16:05:59.531198 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:0
--local_ip_for_outbound_sockets=127.25.109.1
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:38619
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:59.661113 11972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:59.661381 11972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:59.661464 11972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:59.665675 11972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:59.665733 11972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:59.665836 11972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:05:59.671115 11972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:38619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:05:59.672237 11972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:59.673372 11972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:59.683429 11977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:59.683492 11980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:59.683532 11972 server_base.cc:1034] running on GCE node
W20260924 16:05:59.683430 11978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:59.684144 11972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:59.684854 11972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:59.686043 11972 hybrid_clock.cc:648] HybridClock initialized: now 1790265959686023 us; error 35 us; skew 500 ppm
Sep 24 16:05:59 dist-test-slave-n106 krb5kdc[11863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265959, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:59.688894 11972 init.cc:377] Logged in from keytab as kudu/127.25.109.1@KRBTEST.COM (short username kudu)
I20260924 16:05:59.689749 11972 webserver.cc:460] Webserver started at http://127.25.109.1:37601/ using document root <none> and password file <none>
I20260924 16:05:59.690327 11972 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:59.690418 11972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:59.690654 11972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:59.692796 11972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-0/data/instance:
uuid: "342864b0d69444c28e572094a115a662"
format_stamp: "Formatted at 2026-09-24 16:05:59 on dist-test-slave-n106"
server_key: "f7d195753bdd89009fe21dcecf03c8e7"
server_key_iv: "4ea05a0055b20e733075340fee76aa18"
server_key_version: "encryptionkey@0"
I20260924 16:05:59.693380 11972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance:
uuid: "342864b0d69444c28e572094a115a662"
format_stamp: "Formatted at 2026-09-24 16:05:59 on dist-test-slave-n106"
server_key: "f7d195753bdd89009fe21dcecf03c8e7"
server_key_iv: "4ea05a0055b20e733075340fee76aa18"
server_key_version: "encryptionkey@0"
I20260924 16:05:59.697090 11972 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:05:59.699713 11987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:59.700985 11972 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:05:59.701126 11972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "342864b0d69444c28e572094a115a662"
format_stamp: "Formatted at 2026-09-24 16:05:59 on dist-test-slave-n106"
server_key: "f7d195753bdd89009fe21dcecf03c8e7"
server_key_iv: "4ea05a0055b20e733075340fee76aa18"
server_key_version: "encryptionkey@0"
I20260924 16:05:59.701238 11972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:59.723065 11972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:59.726684 11972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:59.726903 11972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:59.727536 11972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:05:59.728473 11972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:59.728547 11972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:59.728619 11972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:59.728670 11972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:59.738098 11972 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:46205
I20260924 16:05:59.738117 12101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:46205 every 8 connection(s)
I20260924 16:05:59.739121 11972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:05:59.748780 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 11972
I20260924 16:05:59.748944 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance
Sep 24 16:05:59 dist-test-slave-n106 krb5kdc[11863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265959, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:05:59.749338 26036 external_mini_cluster.cc:1428] Setting key ddfbbf5f11f7a32ab5c837e4e529e2cd
I20260924 16:05:59.753974 11948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:59.741100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:52857 (local address 127.25.109.62:38619)
0924 16:05:59.741248 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:59.741252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:59.742391 (+ 1139us) server_negotiation.cc:407] Connection header received
0924 16:05:59.743363 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:59.743367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:59.743443 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:59.743547 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:59.744845 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:59.745401 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:59.746581 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:59.746839 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:59.746963 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:59.749834 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:59.749839 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:59.749843 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:59.749875 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:59.751373 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:59.752030 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:59.752033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:59.752034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:59.752114 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:59.752635 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:59.752642 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:59.752645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:59.752928 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:59.753057 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:59.753586 (+ 529us) server_negotiation.cc:299] Negotiation successful
0924 16:05:59.753755 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 16:05:59.754695 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:59.741511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:38619 (local address 127.25.109.1:52857)
0924 16:05:59.742229 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:59.742273 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:59.743145 (+ 872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:59.743721 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:59.743729 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:59.744111 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:59.744692 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:59.744710 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:59.745548 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:59.745551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:59.746437 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:59.746454 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:59.747001 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:59.747005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:59.747187 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:59.747804 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:59.747821 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:59.749623 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:59.751500 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:59.751510 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:59.751529 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:59.751919 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:59.752210 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:59.752216 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:59.752221 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:59.752448 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:59.753057 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:59.753063 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:59.753313 (+ 250us) client_negotiation.cc:769] Sending connection context
0924 16:05:59.753472 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 16:05:59.753750 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":114,"threads_started":1}
I20260924 16:05:59.756036 12102 heartbeater.cc:347] Connected to a master server at 127.25.109.62:38619
I20260924 16:05:59.756300 12102 heartbeater.cc:464] Registering TS with master...
I20260924 16:05:59.756837 12102 heartbeater.cc:511] Master 127.25.109.62:38619 requested a full tablet report, sending...
I20260924 16:05:59.758396 11908 ts_manager.cc:194] Registered new tserver with Master: 342864b0d69444c28e572094a115a662 (127.25.109.1:46205)
WARNING: no policy specified for kudu/127.25.109.2@KRBTEST.COM; defaulting to no policy
I20260924 16:05:59.759616 11908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.1@KRBTEST.COM'} at 127.25.109.1:52857
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
I20260924 16:05:59.777490 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:0
--local_ip_for_outbound_sockets=127.25.109.2
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:38619
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:59.891103 12107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:59.891357 12107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:59.891428 12107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:59.895681 12107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:59.895738 12107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:59.895830 12107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:05:59.901399 12107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:38619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:05:59.902482 12107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:59.903774 12107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:59.914129 12113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:59.914119 12112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:59.914119 12115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:59.914300 12107 server_base.cc:1034] running on GCE node
I20260924 16:05:59.914713 12107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:59.915380 12107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:59.916565 12107 hybrid_clock.cc:648] HybridClock initialized: now 1790265959916541 us; error 35 us; skew 500 ppm
Sep 24 16:05:59 dist-test-slave-n106 krb5kdc[11863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265959, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:59.924381 12107 init.cc:377] Logged in from keytab as kudu/127.25.109.2@KRBTEST.COM (short username kudu)
I20260924 16:05:59.925390 12107 webserver.cc:460] Webserver started at http://127.25.109.2:36213/ using document root <none> and password file <none>
I20260924 16:05:59.926011 12107 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:59.926083 12107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:59.926257 12107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:59.928562 12107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-1/data/instance:
uuid: "864803cae41b4b13a6d625c01c48e22e"
format_stamp: "Formatted at 2026-09-24 16:05:59 on dist-test-slave-n106"
server_key: "a10de6336ac151e0b68a84d0810ade3f"
server_key_iv: "06b4d25c03af0ccdcc34f67cc2f9998c"
server_key_version: "encryptionkey@0"
I20260924 16:05:59.929121 12107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance:
uuid: "864803cae41b4b13a6d625c01c48e22e"
format_stamp: "Formatted at 2026-09-24 16:05:59 on dist-test-slave-n106"
server_key: "a10de6336ac151e0b68a84d0810ade3f"
server_key_iv: "06b4d25c03af0ccdcc34f67cc2f9998c"
server_key_version: "encryptionkey@0"
I20260924 16:05:59.933167 12107 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:05:59.935842 12122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:59.937250 12107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:05:59.937361 12107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "864803cae41b4b13a6d625c01c48e22e"
format_stamp: "Formatted at 2026-09-24 16:05:59 on dist-test-slave-n106"
server_key: "a10de6336ac151e0b68a84d0810ade3f"
server_key_iv: "06b4d25c03af0ccdcc34f67cc2f9998c"
server_key_version: "encryptionkey@0"
I20260924 16:05:59.937516 12107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:59.948251 12107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:59.951675 12107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:59.951887 12107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:59.952562 12107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:05:59.953423 12107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:59.953469 12107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:59.953540 12107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:59.953579 12107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:59.962466 12107 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:33335
I20260924 16:05:59.962517 12237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:33335 every 8 connection(s)
I20260924 16:05:59.963426 12107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:05:59.964017 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 12107
I20260924 16:05:59.964164 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance
I20260924 16:05:59.964564 26036 external_mini_cluster.cc:1428] Setting key 8b27cc1940eb7bca9ca0aefaab20f415
Sep 24 16:05:59 dist-test-slave-n106 krb5kdc[11863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265959, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.3@KRBTEST.COM" created.
I20260924 16:05:59.978273 11948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:59.965632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:54127 (local address 127.25.109.62:38619)
0924 16:05:59.965766 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:59.965769 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:59.967114 (+ 1345us) server_negotiation.cc:407] Connection header received
0924 16:05:59.968048 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:59.968051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:59.968102 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:59.968172 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:59.969714 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:59.970603 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:59.971725 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:59.972173 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:59.972307 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:59.974835 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:59.974842 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:59.974848 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:59.974885 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:59.976176 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:59.976751 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:59.976754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:59.976755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:59.976830 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:59.977173 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:59.977175 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:59.977176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:59.977351 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:59.977494 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:59.977979 (+ 485us) server_negotiation.cc:299] Negotiation successful
0924 16:05:59.978099 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 16:05:59.979267 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:59.966169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:38619 (local address 127.25.109.2:54127)
0924 16:05:59.966941 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:59.966993 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:59.967804 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:59.968363 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:59.968374 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:59.968918 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:59.969544 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:59.969560 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:59.970730 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:59.970733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:59.971587 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:59.971596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:59.972321 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:59.972324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:59.972531 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:59.973225 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:59.973241 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:59.974656 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:59.976279 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:59.976284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:59.976295 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:59.976658 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:59.976920 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:59.976923 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:59.976924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:59.977060 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:59.977493 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:59.977500 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:59.977739 (+ 239us) client_negotiation.cc:769] Sending connection context
0924 16:05:59.977922 (+ 183us) client_negotiation.cc:240] Negotiation successful
0924 16:05:59.978251 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":159,"threads_started":1}
I20260924 16:05:59.980594 12238 heartbeater.cc:347] Connected to a master server at 127.25.109.62:38619
I20260924 16:05:59.980861 12238 heartbeater.cc:464] Registering TS with master...
I20260924 16:05:59.981504 12238 heartbeater.cc:511] Master 127.25.109.62:38619 requested a full tablet report, sending...
I20260924 16:05:59.982748 11908 ts_manager.cc:194] Registered new tserver with Master: 864803cae41b4b13a6d625c01c48e22e (127.25.109.2:33335)
I20260924 16:05:59.983331 11908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.2@KRBTEST.COM'} at 127.25.109.2:54127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
I20260924 16:05:59.995779 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:0
--local_ip_for_outbound_sockets=127.25.109.3
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:38619
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:00.109735 12243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:00.110039 12243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:00.110117 12243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:00.114197 12243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:00.114282 12243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:00.114373 12243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:06:00.119603 12243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:38619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:00.120766 12243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:00.121829 12243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:00.131609 12251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:00.131631 12248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:00.131595 12249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:00.131886 12243 server_base.cc:1034] running on GCE node
I20260924 16:06:00.132305 12243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:00.132987 12243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:00.134150 12243 hybrid_clock.cc:648] HybridClock initialized: now 1790265960134130 us; error 28 us; skew 500 ppm
Sep 24 16:06:00 dist-test-slave-n106 krb5kdc[11863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265960, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:00.137064 12243 init.cc:377] Logged in from keytab as kudu/127.25.109.3@KRBTEST.COM (short username kudu)
I20260924 16:06:00.137923 12243 webserver.cc:460] Webserver started at http://127.25.109.3:40343/ using document root <none> and password file <none>
I20260924 16:06:00.138450 12243 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:00.138506 12243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:00.138681 12243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:00.140805 12243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-2/data/instance:
uuid: "3ad00b5eebab42a8b522a3289097fe7e"
format_stamp: "Formatted at 2026-09-24 16:06:00 on dist-test-slave-n106"
server_key: "2afdb9c5e09a96dc52db990d8fd576a0"
server_key_iv: "4304166a93b3fd2c1bc7651553830ef6"
server_key_version: "encryptionkey@0"
I20260924 16:06:00.141381 12243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance:
uuid: "3ad00b5eebab42a8b522a3289097fe7e"
format_stamp: "Formatted at 2026-09-24 16:06:00 on dist-test-slave-n106"
server_key: "2afdb9c5e09a96dc52db990d8fd576a0"
server_key_iv: "4304166a93b3fd2c1bc7651553830ef6"
server_key_version: "encryptionkey@0"
I20260924 16:06:00.145314 12243 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:06:00.147784 12258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:00.149121 12243 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:06:00.149222 12243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "3ad00b5eebab42a8b522a3289097fe7e"
format_stamp: "Formatted at 2026-09-24 16:06:00 on dist-test-slave-n106"
server_key: "2afdb9c5e09a96dc52db990d8fd576a0"
server_key_iv: "4304166a93b3fd2c1bc7651553830ef6"
server_key_version: "encryptionkey@0"
I20260924 16:06:00.149313 12243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:00.161391 12243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:00.164887 12243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:00.165113 12243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:00.165683 12243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:06:00.166534 12243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:00.166581 12243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:00.166631 12243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:00.166653 12243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:00.175568 12372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:36847 every 8 connection(s)
I20260924 16:06:00.175547 12243 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:36847
I20260924 16:06:00.176719 12243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:06:00.182448 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 12243
I20260924 16:06:00.182613 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance
I20260924 16:06:00.183013 26036 external_mini_cluster.cc:1428] Setting key 00d793efcab0bcf678f1b327a5ff5c8a
Sep 24 16:06:00 dist-test-slave-n106 krb5kdc[11863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265960, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:06:00.190898 11948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:00.178672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:47309 (local address 127.25.109.62:38619)
0924 16:06:00.178822 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:00.178826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:00.179727 (+ 901us) server_negotiation.cc:407] Connection header received
0924 16:06:00.181152 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:00.181156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:00.181227 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:00.181331 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:00.182575 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.183236 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.184755 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.184986 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.185112 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:00.187605 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:00.187609 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:00.187611 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:00.187636 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:00.189001 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:00.189493 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:00.189496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:00.189497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:00.189576 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:00.189907 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:00.189910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:00.189911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:00.190105 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:00.190183 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:00.190663 (+ 480us) server_negotiation.cc:299] Negotiation successful
0924 16:06:00.190770 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:06:00.191773 12375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:00.179075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:38619 (local address 127.25.109.3:47309)
0924 16:06:00.179567 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:00.179617 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:00.180910 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:00.181532 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:00.181540 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:00.181922 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:00.182423 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:00.182437 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.183372 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.183375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:00.184633 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:00.184643 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.185145 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.185150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:00.185337 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:00.185873 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:00.185889 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:00.187432 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:00.189114 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:00.189121 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:00.189136 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:00.189405 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:00.189673 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:00.189676 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:00.189677 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:00.189824 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:00.190194 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:00.190200 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:00.190447 (+ 247us) client_negotiation.cc:769] Sending connection context
0924 16:06:00.190610 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 16:06:00.190894 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":115,"threads_started":1}
I20260924 16:06:00.193054 12373 heartbeater.cc:347] Connected to a master server at 127.25.109.62:38619
I20260924 16:06:00.193346 12373 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:00.194123 12373 heartbeater.cc:511] Master 127.25.109.62:38619 requested a full tablet report, sending...
I20260924 16:06:00.195348 11908 ts_manager.cc:194] Registered new tserver with Master: 3ad00b5eebab42a8b522a3289097fe7e (127.25.109.3:36847)
I20260924 16:06:00.196012 11908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.3@KRBTEST.COM'} at 127.25.109.3:47309
I20260924 16:06:00.197204 26036 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:06:00.207306 11948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:00.199473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39556 (local address 127.25.109.62:38619)
0924 16:06:00.199599 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:00.199602 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:00.199835 (+ 233us) server_negotiation.cc:407] Connection header received
0924 16:06:00.199934 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:00.199936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:00.199976 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:00.200049 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:00.200847 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.201353 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.202193 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.202411 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.202499 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:00.204056 (+ 1557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:00.204059 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:00.204061 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:00.204086 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:00.205691 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:00.206186 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:00.206188 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:00.206190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:00.206267 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:00.206546 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:00.206549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:00.206550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:00.206771 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:00.206887 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:00.207104 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 16:06:00.207181 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 16:06:00.210510 11908 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:06:00.214923 11908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:06:00.230398 12384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:00.223922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:36847 (local address 127.0.0.1:47120)
0924 16:06:00.224308 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:00.224359 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:00.224527 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:00.225340 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:00.225347 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:00.225372 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:00.225523 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:00.225534 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.226995 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.227001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:00.228921 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:00.228932 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.229775 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.229779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:00.229887 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:00.229926 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 16:06:00.230085 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 16:06:00.230220 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":139,"thread_start_us":104,"threads_started":1}
I20260924 16:06:00.230393 12386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:00.224086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47120 (local address 127.25.109.3:36847)
0924 16:06:00.224776 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:00.224792 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:00.224818 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:06:00.224904 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:00.224912 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:00.225143 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:00.225703 (+ 560us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:00.225862 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.226871 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.229052 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.229644 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.229773 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:00.229917 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:00.230125 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 16:06:00.230225 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":120,"threads_started":1}
I20260924 16:06:00.230505 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:00.225284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56614 (local address 127.25.109.2:33335)
0924 16:06:00.226014 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:00.226025 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:00.226046 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:06:00.226114 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:00.226119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:00.226292 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:00.226439 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:00.226677 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.227558 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.228847 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.229651 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.229758 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:00.229909 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:00.230274 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 16:06:00.230373 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":99,"threads_started":1}
I20260924 16:06:00.231014 12387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:00.224690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44570 (local address 127.25.109.1:46205)
0924 16:06:00.224963 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:00.224978 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:00.225006 (+ 28us) server_negotiation.cc:407] Connection header received
0924 16:06:00.225091 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:00.225096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:00.225306 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:00.225473 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:00.225927 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.227047 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.228791 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.229655 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.229758 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:00.229909 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:00.230827 (+ 918us) server_negotiation.cc:299] Negotiation successful
0924 16:06:00.230907 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":115,"threads_started":1}
I20260924 16:06:00.230448 12385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:00.224070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:46205 (local address 127.0.0.1:44570)
0924 16:06:00.224755 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:00.224770 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:00.224867 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:00.225649 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:00.225653 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:00.225670 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:00.225790 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:00.225799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.227198 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.227201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:00.228659 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:00.228672 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.229975 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.229979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:00.230091 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:00.230124 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 16:06:00.230196 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 16:06:00.230291 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":102,"thread_start_us":104,"threads_started":1}
I20260924 16:06:00.231392 12388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:00.225123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:33335 (local address 127.0.0.1:56614)
0924 16:06:00.225666 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:00.225679 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:00.225757 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:00.226442 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:00.226445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:00.226465 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:00.226563 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:00.226568 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.227693 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.227698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:00.228690 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:00.228700 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.229975 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.229979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:00.230096 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:00.230132 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 16:06:00.230196 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 16:06:00.230751 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":549,"thread_start_us":96,"threads_started":1}
I20260924 16:06:00.233881 12307 tablet_service.cc:1467] Processing CreateTablet for tablet 2893bf6e42d9471d8fe705ed3b2483e0 (DEFAULT_TABLE table=test-table [id=170ca2c9499842b7955bdb63a6a6b27f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:00.234246 12036 tablet_service.cc:1467] Processing CreateTablet for tablet 2893bf6e42d9471d8fe705ed3b2483e0 (DEFAULT_TABLE table=test-table [id=170ca2c9499842b7955bdb63a6a6b27f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:00.234472 12170 tablet_service.cc:1467] Processing CreateTablet for tablet 2893bf6e42d9471d8fe705ed3b2483e0 (DEFAULT_TABLE table=test-table [id=170ca2c9499842b7955bdb63a6a6b27f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:00.234961 12307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2893bf6e42d9471d8fe705ed3b2483e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:00.235349 12170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2893bf6e42d9471d8fe705ed3b2483e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:00.235349 12036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2893bf6e42d9471d8fe705ed3b2483e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:00.242148 12390 tablet_bootstrap.cc:492] T 2893bf6e42d9471d8fe705ed3b2483e0 P 3ad00b5eebab42a8b522a3289097fe7e: Bootstrap starting.
I20260924 16:06:00.242915 12392 tablet_bootstrap.cc:492] T 2893bf6e42d9471d8fe705ed3b2483e0 P 864803cae41b4b13a6d625c01c48e22e: Bootstrap starting.
I20260924 16:06:00.243688 12391 tablet_bootstrap.cc:492] T 2893bf6e42d9471d8fe705ed3b2483e0 P 342864b0d69444c28e572094a115a662: Bootstrap starting.
I20260924 16:06:00.244980 12390 tablet_bootstrap.cc:654] T 2893bf6e42d9471d8fe705ed3b2483e0 P 3ad00b5eebab42a8b522a3289097fe7e: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:00.245463 12392 tablet_bootstrap.cc:654] T 2893bf6e42d9471d8fe705ed3b2483e0 P 864803cae41b4b13a6d625c01c48e22e: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:00.245702 12391 tablet_bootstrap.cc:654] T 2893bf6e42d9471d8fe705ed3b2483e0 P 342864b0d69444c28e572094a115a662: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:00.246448 12392 log.cc:824] T 2893bf6e42d9471d8fe705ed3b2483e0 P 864803cae41b4b13a6d625c01c48e22e: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:00.246448 12390 log.cc:824] T 2893bf6e42d9471d8fe705ed3b2483e0 P 3ad00b5eebab42a8b522a3289097fe7e: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:00.247009 12391 log.cc:824] T 2893bf6e42d9471d8fe705ed3b2483e0 P 342864b0d69444c28e572094a115a662: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:00.248709 12390 tablet_bootstrap.cc:492] T 2893bf6e42d9471d8fe705ed3b2483e0 P 3ad00b5eebab42a8b522a3289097fe7e: No bootstrap required, opened a new log
I20260924 16:06:00.248826 12392 tablet_bootstrap.cc:492] T 2893bf6e42d9471d8fe705ed3b2483e0 P 864803cae41b4b13a6d625c01c48e22e: No bootstrap required, opened a new log
I20260924 16:06:00.248927 12390 ts_tablet_manager.cc:1397] T 2893bf6e42d9471d8fe705ed3b2483e0 P 3ad00b5eebab42a8b522a3289097fe7e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260924 16:06:00.248987 12391 tablet_bootstrap.cc:492] T 2893bf6e42d9471d8fe705ed3b2483e0 P 342864b0d69444c28e572094a115a662: No bootstrap required, opened a new log
I20260924 16:06:00.249065 12392 ts_tablet_manager.cc:1397] T 2893bf6e42d9471d8fe705ed3b2483e0 P 864803cae41b4b13a6d625c01c48e22e: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260924 16:06:00.249171 12391 ts_tablet_manager.cc:1397] T 2893bf6e42d9471d8fe705ed3b2483e0 P 342864b0d69444c28e572094a115a662: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260924 16:06:00.252184 12392 raft_consensus.cc:348] T 2893bf6e42d9471d8fe705ed3b2483e0 P 864803cae41b4b13a6d625c01c48e22e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad00b5eebab42a8b522a3289097fe7e" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 36847 } } peers { permanent_uuid: "342864b0d69444c28e572094a115a662" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46205 } } peers { permanent_uuid: "864803cae41b4b13a6d625c01c48e22e" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 33335 } }
I20260924 16:06:00.252525 12392 raft_consensus.cc:374] T 2893bf6e42d9471d8fe705ed3b2483e0 P 864803cae41b4b13a6d625c01c48e22e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:00.252593 12392 raft_consensus.cc:729] T 2893bf6e42d9471d8fe705ed3b2483e0 P 864803cae41b4b13a6d625c01c48e22e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 864803cae41b4b13a6d625c01c48e22e, State: Initialized, Role: FOLLOWER
I20260924 16:06:00.252941 12390 raft_consensus.cc:348] T 2893bf6e42d9471d8fe705ed3b2483e0 P 3ad00b5eebab42a8b522a3289097fe7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad00b5eebab42a8b522a3289097fe7e" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 36847 } } peers { permanent_uuid: "342864b0d69444c28e572094a115a662" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46205 } } peers { permanent_uuid: "864803cae41b4b13a6d625c01c48e22e" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 33335 } }
I20260924 16:06:00.252941 12391 raft_consensus.cc:348] T 2893bf6e42d9471d8fe705ed3b2483e0 P 342864b0d69444c28e572094a115a662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad00b5eebab42a8b522a3289097fe7e" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 36847 } } peers { permanent_uuid: "342864b0d69444c28e572094a115a662" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46205 } } peers { permanent_uuid: "864803cae41b4b13a6d625c01c48e22e" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 33335 } }
I20260924 16:06:00.253207 12391 raft_consensus.cc:374] T 2893bf6e42d9471d8fe705ed3b2483e0 P 342864b0d69444c28e572094a115a662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:00.253207 12390 raft_consensus.cc:374] T 2893bf6e42d9471d8fe705ed3b2483e0 P 3ad00b5eebab42a8b522a3289097fe7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:00.253266 12390 raft_consensus.cc:729] T 2893bf6e42d9471d8fe705ed3b2483e0 P 3ad00b5eebab42a8b522a3289097fe7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ad00b5eebab42a8b522a3289097fe7e, State: Initialized, Role: FOLLOWER
I20260924 16:06:00.253266 12391 raft_consensus.cc:729] T 2893bf6e42d9471d8fe705ed3b2483e0 P 342864b0d69444c28e572094a115a662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342864b0d69444c28e572094a115a662, State: Initialized, Role: FOLLOWER
I20260924 16:06:00.253247 12392 consensus_queue.cc:260] T 2893bf6e42d9471d8fe705ed3b2483e0 P 864803cae41b4b13a6d625c01c48e22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad00b5eebab42a8b522a3289097fe7e" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 36847 } } peers { permanent_uuid: "342864b0d69444c28e572094a115a662" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46205 } } peers { permanent_uuid: "864803cae41b4b13a6d625c01c48e22e" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 33335 } }
I20260924 16:06:00.253697 12391 consensus_queue.cc:260] T 2893bf6e42d9471d8fe705ed3b2483e0 P 342864b0d69444c28e572094a115a662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad00b5eebab42a8b522a3289097fe7e" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 36847 } } peers { permanent_uuid: "342864b0d69444c28e572094a115a662" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46205 } } peers { permanent_uuid: "864803cae41b4b13a6d625c01c48e22e" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 33335 } }
I20260924 16:06:00.253697 12390 consensus_queue.cc:260] T 2893bf6e42d9471d8fe705ed3b2483e0 P 3ad00b5eebab42a8b522a3289097fe7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad00b5eebab42a8b522a3289097fe7e" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 36847 } } peers { permanent_uuid: "342864b0d69444c28e572094a115a662" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46205 } } peers { permanent_uuid: "864803cae41b4b13a6d625c01c48e22e" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 33335 } }
I20260924 16:06:00.254136 12238 heartbeater.cc:503] Master 127.25.109.62:38619 was elected leader, sending a full tablet report...
I20260924 16:06:00.254714 12391 ts_tablet_manager.cc:1428] T 2893bf6e42d9471d8fe705ed3b2483e0 P 342864b0d69444c28e572094a115a662: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260924 16:06:00.254714 12390 ts_tablet_manager.cc:1428] T 2893bf6e42d9471d8fe705ed3b2483e0 P 3ad00b5eebab42a8b522a3289097fe7e: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260924 16:06:00.255091 12373 heartbeater.cc:503] Master 127.25.109.62:38619 was elected leader, sending a full tablet report...
I20260924 16:06:00.255497 12102 heartbeater.cc:503] Master 127.25.109.62:38619 was elected leader, sending a full tablet report...
I20260924 16:06:00.255759 12392 ts_tablet_manager.cc:1428] T 2893bf6e42d9471d8fe705ed3b2483e0 P 864803cae41b4b13a6d625c01c48e22e: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20260924 16:06:00.427788 12374 tablet.cc:2369] T 2893bf6e42d9471d8fe705ed3b2483e0 P 3ad00b5eebab42a8b522a3289097fe7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:06:00.464752 12239 tablet.cc:2369] T 2893bf6e42d9471d8fe705ed3b2483e0 P 864803cae41b4b13a6d625c01c48e22e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:06:00.490835 12103 tablet.cc:2369] T 2893bf6e42d9471d8fe705ed3b2483e0 P 342864b0d69444c28e572094a115a662: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:06:00.556038 12396 raft_consensus.cc:482] T 2893bf6e42d9471d8fe705ed3b2483e0 P 864803cae41b4b13a6d625c01c48e22e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:06:00.556270 12396 raft_consensus.cc:504] T 2893bf6e42d9471d8fe705ed3b2483e0 P 864803cae41b4b13a6d625c01c48e22e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad00b5eebab42a8b522a3289097fe7e" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 36847 } } peers { permanent_uuid: "342864b0d69444c28e572094a115a662" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46205 } } peers { permanent_uuid: "864803cae41b4b13a6d625c01c48e22e" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 33335 } }
I20260924 16:06:00.557575 12396 leader_election.cc:288] T 2893bf6e42d9471d8fe705ed3b2483e0 P 864803cae41b4b13a6d625c01c48e22e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ad00b5eebab42a8b522a3289097fe7e (127.25.109.3:36847), 342864b0d69444c28e572094a115a662 (127.25.109.1:46205)
I20260924 16:06:00.561628 12399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:00.557810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:36847 (local address 127.25.109.2:41703)
0924 16:06:00.558133 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:00.558149 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:00.558269 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:00.558592 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:00.558596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:00.558615 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:00.558760 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:00.558767 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.559592 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.559596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:00.560486 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:00.560495 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.561187 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.561191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:00.561285 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:00.561316 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 16:06:00.561386 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 16:06:00.561442 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":96,"threads_started":1}
I20260924 16:06:00.561645 12386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:00.557925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:41703 (local address 127.25.109.3:36847)
0924 16:06:00.558180 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:00.558186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:00.558213 (+ 27us) server_negotiation.cc:407] Connection header received
0924 16:06:00.558406 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:00.558413 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:00.558475 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:00.558556 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:00.558878 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.559475 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.560612 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.561048 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.561156 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:00.561241 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:00.561423 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 16:06:00.561510 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260924 16:06:00.561679 12400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:00.557914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:46205 (local address 127.25.109.2:52723)
0924 16:06:00.558366 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:00.558380 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:00.558489 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:00.558786 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:00.558788 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:00.558803 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:00.558961 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:00.558967 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.559704 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.559707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:00.560537 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:00.560567 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.561238 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.561241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:00.561353 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:00.561397 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 16:06:00.561476 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 16:06:00.561541 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":105,"threads_started":1}
I20260924 16:06:00.561725 12387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:00.557998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:52723 (local address 127.25.109.1:46205)
0924 16:06:00.558150 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:00.558157 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:00.558478 (+ 321us) server_negotiation.cc:407] Connection header received
0924 16:06:00.558601 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:00.558604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:00.558676 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:00.558762 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:00.559074 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.559586 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.560701 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.561084 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.561226 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:00.561282 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:00.561513 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 16:06:00.561587 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 16:06:00.562335 12056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2893bf6e42d9471d8fe705ed3b2483e0" candidate_uuid: "864803cae41b4b13a6d625c01c48e22e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342864b0d69444c28e572094a115a662" is_pre_election: true
I20260924 16:06:00.562328 12327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2893bf6e42d9471d8fe705ed3b2483e0" candidate_uuid: "864803cae41b4b13a6d625c01c48e22e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ad00b5eebab42a8b522a3289097fe7e" is_pre_election: true
I20260924 16:06:00.562668 12056 raft_consensus.cc:2454] T 2893bf6e42d9471d8fe705ed3b2483e0 P 342864b0d69444c28e572094a115a662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 864803cae41b4b13a6d625c01c48e22e in term 0.
I20260924 16:06:00.562668 12327 raft_consensus.cc:2454] T 2893bf6e42d9471d8fe705ed3b2483e0 P 3ad00b5eebab42a8b522a3289097fe7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 864803cae41b4b13a6d625c01c48e22e in term 0.
I20260924 16:06:00.563202 12126 leader_election.cc:302] T 2893bf6e42d9471d8fe705ed3b2483e0 P 864803cae41b4b13a6d625c01c48e22e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 342864b0d69444c28e572094a115a662, 864803cae41b4b13a6d625c01c48e22e; no voters:
I20260924 16:06:00.563482 12396 raft_consensus.cc:2781] T 2893bf6e42d9471d8fe705ed3b2483e0 P 864803cae41b4b13a6d625c01c48e22e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:06:00.563570 12396 raft_consensus.cc:482] T 2893bf6e42d9471d8fe705ed3b2483e0 P 864803cae41b4b13a6d625c01c48e22e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:06:00.563632 12396 raft_consensus.cc:3037] T 2893bf6e42d9471d8fe705ed3b2483e0 P 864803cae41b4b13a6d625c01c48e22e [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:00.564913 12396 raft_consensus.cc:504] T 2893bf6e42d9471d8fe705ed3b2483e0 P 864803cae41b4b13a6d625c01c48e22e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad00b5eebab42a8b522a3289097fe7e" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 36847 } } peers { permanent_uuid: "342864b0d69444c28e572094a115a662" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46205 } } peers { permanent_uuid: "864803cae41b4b13a6d625c01c48e22e" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 33335 } }
I20260924 16:06:00.565272 12396 leader_election.cc:288] T 2893bf6e42d9471d8fe705ed3b2483e0 P 864803cae41b4b13a6d625c01c48e22e [CANDIDATE]: Term 1 election: Requested vote from peers 3ad00b5eebab42a8b522a3289097fe7e (127.25.109.3:36847), 342864b0d69444c28e572094a115a662 (127.25.109.1:46205)
I20260924 16:06:00.565645 12056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2893bf6e42d9471d8fe705ed3b2483e0" candidate_uuid: "864803cae41b4b13a6d625c01c48e22e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342864b0d69444c28e572094a115a662"
I20260924 16:06:00.565653 12327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2893bf6e42d9471d8fe705ed3b2483e0" candidate_uuid: "864803cae41b4b13a6d625c01c48e22e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ad00b5eebab42a8b522a3289097fe7e"
I20260924 16:06:00.565788 12056 raft_consensus.cc:3037] T 2893bf6e42d9471d8fe705ed3b2483e0 P 342864b0d69444c28e572094a115a662 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:00.565800 12327 raft_consensus.cc:3037] T 2893bf6e42d9471d8fe705ed3b2483e0 P 3ad00b5eebab42a8b522a3289097fe7e [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:00.566849 12056 raft_consensus.cc:2454] T 2893bf6e42d9471d8fe705ed3b2483e0 P 342864b0d69444c28e572094a115a662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 864803cae41b4b13a6d625c01c48e22e in term 1.
I20260924 16:06:00.566874 12327 raft_consensus.cc:2454] T 2893bf6e42d9471d8fe705ed3b2483e0 P 3ad00b5eebab42a8b522a3289097fe7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 864803cae41b4b13a6d625c01c48e22e in term 1.
I20260924 16:06:00.567174 12126 leader_election.cc:302] T 2893bf6e42d9471d8fe705ed3b2483e0 P 864803cae41b4b13a6d625c01c48e22e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 342864b0d69444c28e572094a115a662, 864803cae41b4b13a6d625c01c48e22e; no voters:
I20260924 16:06:00.567348 12396 raft_consensus.cc:2781] T 2893bf6e42d9471d8fe705ed3b2483e0 P 864803cae41b4b13a6d625c01c48e22e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:00.567559 12396 raft_consensus.cc:686] T 2893bf6e42d9471d8fe705ed3b2483e0 P 864803cae41b4b13a6d625c01c48e22e [term 1 LEADER]: Becoming Leader. State: Replica: 864803cae41b4b13a6d625c01c48e22e, State: Running, Role: LEADER
I20260924 16:06:00.567903 12396 consensus_queue.cc:237] T 2893bf6e42d9471d8fe705ed3b2483e0 P 864803cae41b4b13a6d625c01c48e22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad00b5eebab42a8b522a3289097fe7e" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 36847 } } peers { permanent_uuid: "342864b0d69444c28e572094a115a662" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46205 } } peers { permanent_uuid: "864803cae41b4b13a6d625c01c48e22e" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 33335 } }
I20260924 16:06:00.571354 11907 catalog_manager.cc:5495] T 2893bf6e42d9471d8fe705ed3b2483e0 P 864803cae41b4b13a6d625c01c48e22e reported cstate change: term changed from 0 to 1, leader changed from <none> to 864803cae41b4b13a6d625c01c48e22e (127.25.109.2). New cstate: current_term: 1 leader_uuid: "864803cae41b4b13a6d625c01c48e22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad00b5eebab42a8b522a3289097fe7e" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 36847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "342864b0d69444c28e572094a115a662" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "864803cae41b4b13a6d625c01c48e22e" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 33335 } health_report { overall_health: HEALTHY } } }
I20260924 16:06:00.602468 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:00.598965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56630 (local address 127.25.109.2:33335)
0924 16:06:00.599096 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:00.599100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:00.599115 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:06:00.599222 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:00.599224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:00.599283 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:00.599372 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:06:00.599657 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.600171 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.601094 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.601311 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.601394 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:00.601683 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:06:00.602122 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:06:00.602235 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:00.602311 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 16:06:00.602350 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 16:06:00.609119 12327 raft_consensus.cc:1261] T 2893bf6e42d9471d8fe705ed3b2483e0 P 3ad00b5eebab42a8b522a3289097fe7e [term 1 FOLLOWER]: Refusing update from remote peer 864803cae41b4b13a6d625c01c48e22e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:06:00.609112 12056 raft_consensus.cc:1261] T 2893bf6e42d9471d8fe705ed3b2483e0 P 342864b0d69444c28e572094a115a662 [term 1 FOLLOWER]: Refusing update from remote peer 864803cae41b4b13a6d625c01c48e22e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:06:00.609825 12401 consensus_queue.cc:1035] T 2893bf6e42d9471d8fe705ed3b2483e0 P 864803cae41b4b13a6d625c01c48e22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ad00b5eebab42a8b522a3289097fe7e" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 36847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:06:00.610026 12396 consensus_queue.cc:1035] T 2893bf6e42d9471d8fe705ed3b2483e0 P 864803cae41b4b13a6d625c01c48e22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "342864b0d69444c28e572094a115a662" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 46205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:06:00.620775 12406 mvcc.cc:205] Tried to move back new op lower bound from 7332929374646505472 to 7332929374488997888. Current Snapshot: MvccSnapshot[applied={T|T < 7332929374646505472}]
I20260924 16:06:00.620972 12405 mvcc.cc:205] Tried to move back new op lower bound from 7332929374646505472 to 7332929374488997888. Current Snapshot: MvccSnapshot[applied={T|T < 7332929374646505472}]
I20260924 16:06:00.626696 11906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:06:00.626855 11906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:06:00.629335 11906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98688153dc664d3ba3a073503b285b20: Sending DeleteTablet for 3 replicas of tablet 2893bf6e42d9471d8fe705ed3b2483e0
I20260924 16:06:00.630170 12036 tablet_service.cc:1514] Processing DeleteTablet for tablet 2893bf6e42d9471d8fe705ed3b2483e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:00 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:44570
I20260924 16:06:00.630167 12307 tablet_service.cc:1514] Processing DeleteTablet for tablet 2893bf6e42d9471d8fe705ed3b2483e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:00 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:47120
I20260924 16:06:00.630697 12414 tablet_replica.cc:330] T 2893bf6e42d9471d8fe705ed3b2483e0 P 3ad00b5eebab42a8b522a3289097fe7e: stopping tablet replica
I20260924 16:06:00.630934 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 11972
I20260924 16:06:00.630951 12170 tablet_service.cc:1514] Processing DeleteTablet for tablet 2893bf6e42d9471d8fe705ed3b2483e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:00 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:56614
I20260924 16:06:00.632915 12414 raft_consensus.cc:2229] T 2893bf6e42d9471d8fe705ed3b2483e0 P 3ad00b5eebab42a8b522a3289097fe7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:06:00.633266 12414 raft_consensus.cc:2258] T 2893bf6e42d9471d8fe705ed3b2483e0 P 3ad00b5eebab42a8b522a3289097fe7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:00.633708 12416 tablet_replica.cc:330] T 2893bf6e42d9471d8fe705ed3b2483e0 P 864803cae41b4b13a6d625c01c48e22e: stopping tablet replica
I20260924 16:06:00.634049 12416 raft_consensus.cc:2229] T 2893bf6e42d9471d8fe705ed3b2483e0 P 864803cae41b4b13a6d625c01c48e22e [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:06:00.634429 12416 raft_consensus.cc:2258] T 2893bf6e42d9471d8fe705ed3b2483e0 P 864803cae41b4b13a6d625c01c48e22e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:00.635093 12414 ts_tablet_manager.cc:1905] T 2893bf6e42d9471d8fe705ed3b2483e0 P 3ad00b5eebab42a8b522a3289097fe7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:00.637964 12416 ts_tablet_manager.cc:1905] T 2893bf6e42d9471d8fe705ed3b2483e0 P 864803cae41b4b13a6d625c01c48e22e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:00.638252 12414 ts_tablet_manager.cc:1918] T 2893bf6e42d9471d8fe705ed3b2483e0 P 3ad00b5eebab42a8b522a3289097fe7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:06:00.638422 12414 log.cc:1196] T 2893bf6e42d9471d8fe705ed3b2483e0 P 3ad00b5eebab42a8b522a3289097fe7e: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265708917108-26036-0/minicluster-data/ts-2/wal/wals/2893bf6e42d9471d8fe705ed3b2483e0
I20260924 16:06:00.638844 12414 ts_tablet_manager.cc:1939] T 2893bf6e42d9471d8fe705ed3b2483e0 P 3ad00b5eebab42a8b522a3289097fe7e: Deleting consensus metadata
W20260924 16:06:00.639633 11895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.109.1:46205 (error 108)
I20260924 16:06:00.640484 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 12107
I20260924 16:06:00.640506 11894 catalog_manager.cc:4841] TS 3ad00b5eebab42a8b522a3289097fe7e (127.25.109.3:36847): tablet 2893bf6e42d9471d8fe705ed3b2483e0 (table test-table [id=170ca2c9499842b7955bdb63a6a6b27f]) successfully deleted
I20260924 16:06:00.642843 12388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:00.642517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:46205 (local address 127.0.0.1:44576)
0924 16:06:00.642673 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:00.642749 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.1:46205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 16:06:00.643035 11895 catalog_manager.cc:4584] TS 342864b0d69444c28e572094a115a662 (127.25.109.1:46205): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2893bf6e42d9471d8fe705ed3b2483e0: Network error: Client connection negotiation failed: client connection to 127.25.109.1:46205: connect: Connection refused (error 111)
I20260924 16:06:00.646929 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 12243
I20260924 16:06:00.648923 12388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:00.647111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:33335 (local address 127.0.0.1:56640)
0924 16:06:00.648729 (+ 1618us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:00.648804 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.2:33335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1493}
I20260924 16:06:00.653257 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 11876
2026-09-24T16:06:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2595 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T16:06:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:00Z Disabled control of system clock
I20260924 16:06:00.684914 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:32885
--webserver_interface=127.25.109.62
--webserver_port=0
--builtin_ntp_servers=127.25.109.20:34489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:32885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 16:06:00.796998 12421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:00.797269 12421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:00.797333 12421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:00.801416 12421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:00.801472 12421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:00.801489 12421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:00.801506 12421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:00.801560 12421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:00.806365 12421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:32885
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:32885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:00.807502 12421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:00.808969 12421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:00.817677 12427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:00.817884 12421 server_base.cc:1034] running on GCE node
W20260924 16:06:00.817791 12426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:00.817677 12429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:00.818583 12421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:00.819624 12421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:00.820784 12421 hybrid_clock.cc:648] HybridClock initialized: now 1790265960820771 us; error 25 us; skew 500 ppm
I20260924 16:06:00.822645 12421 webserver.cc:460] Webserver started at http://127.25.109.62:40947/ using document root <none> and password file <none>
I20260924 16:06:00.823233 12421 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:00.823295 12421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:00.823552 12421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:00.825676 12421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "bebe85ef89da4284aad60d902235b57f"
format_stamp: "Formatted at 2026-09-24 16:06:00 on dist-test-slave-n106"
server_key: "988db5845a70f7c0d1f244beb1d8af17"
server_key_iv: "8235b614b17bcebf75d7f4f84fdc1135"
server_key_version: "encryptionkey@0"
I20260924 16:06:00.826247 12421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "bebe85ef89da4284aad60d902235b57f"
format_stamp: "Formatted at 2026-09-24 16:06:00 on dist-test-slave-n106"
server_key: "988db5845a70f7c0d1f244beb1d8af17"
server_key_iv: "8235b614b17bcebf75d7f4f84fdc1135"
server_key_version: "encryptionkey@0"
I20260924 16:06:00.829811 12421 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:06:00.832224 12435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:00.833477 12421 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:06:00.833606 12421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "bebe85ef89da4284aad60d902235b57f"
format_stamp: "Formatted at 2026-09-24 16:06:00 on dist-test-slave-n106"
server_key: "988db5845a70f7c0d1f244beb1d8af17"
server_key_iv: "8235b614b17bcebf75d7f4f84fdc1135"
server_key_version: "encryptionkey@0"
I20260924 16:06:00.833719 12421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:00.854590 12421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:00.855331 12421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:00.855541 12421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:00.862735 12421 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:32885
I20260924 16:06:00.862739 12487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:32885 every 8 connection(s)
I20260924 16:06:00.863729 12421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:00.866935 12488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:00.871025 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 12421
I20260924 16:06:00.871179 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:06:00.871470 26036 external_mini_cluster.cc:1428] Setting key b2a79fae705addeafbd86e949bf2853d
I20260924 16:06:00.872701 12488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bebe85ef89da4284aad60d902235b57f: Bootstrap starting.
I20260924 16:06:00.874961 12488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bebe85ef89da4284aad60d902235b57f: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:00.875773 12488 log.cc:824] T 00000000000000000000000000000000 P bebe85ef89da4284aad60d902235b57f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:00.877807 12488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bebe85ef89da4284aad60d902235b57f: No bootstrap required, opened a new log
I20260924 16:06:00.881343 12491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:00.872891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42356 (local address 127.25.109.62:32885)
0924 16:06:00.873525 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:00.873540 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:00.873575 (+ 35us) server_negotiation.cc:407] Connection header received
0924 16:06:00.874197 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:00.874214 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:00.874598 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:00.874936 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:00.875370 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.876444 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.877425 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.878088 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.878267 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:00.879090 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:00.879096 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:00.879120 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:00.879615 (+ 495us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:00.879689 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:00.879714 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:00.879815 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:00.880006 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 16:06:00.880288 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":134,"threads_started":1}
I20260924 16:06:00.882437 12488 raft_consensus.cc:348] T 00000000000000000000000000000000 P bebe85ef89da4284aad60d902235b57f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bebe85ef89da4284aad60d902235b57f" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32885 } }
I20260924 16:06:00.882717 12488 raft_consensus.cc:374] T 00000000000000000000000000000000 P bebe85ef89da4284aad60d902235b57f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:00.882778 12488 raft_consensus.cc:729] T 00000000000000000000000000000000 P bebe85ef89da4284aad60d902235b57f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bebe85ef89da4284aad60d902235b57f, State: Initialized, Role: FOLLOWER
I20260924 16:06:00.883242 12488 consensus_queue.cc:260] T 00000000000000000000000000000000 P bebe85ef89da4284aad60d902235b57f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bebe85ef89da4284aad60d902235b57f" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32885 } }
I20260924 16:06:00.883417 12488 raft_consensus.cc:388] T 00000000000000000000000000000000 P bebe85ef89da4284aad60d902235b57f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:00.883492 12488 raft_consensus.cc:482] T 00000000000000000000000000000000 P bebe85ef89da4284aad60d902235b57f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:00.883599 12488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bebe85ef89da4284aad60d902235b57f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:00.884624 12488 raft_consensus.cc:504] T 00000000000000000000000000000000 P bebe85ef89da4284aad60d902235b57f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bebe85ef89da4284aad60d902235b57f" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32885 } }
I20260924 16:06:00.884964 12488 leader_election.cc:302] T 00000000000000000000000000000000 P bebe85ef89da4284aad60d902235b57f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bebe85ef89da4284aad60d902235b57f; no voters:
I20260924 16:06:00.885303 12488 leader_election.cc:288] T 00000000000000000000000000000000 P bebe85ef89da4284aad60d902235b57f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:06:00.885416 12493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bebe85ef89da4284aad60d902235b57f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:00.885658 12493 raft_consensus.cc:686] T 00000000000000000000000000000000 P bebe85ef89da4284aad60d902235b57f [term 1 LEADER]: Becoming Leader. State: Replica: bebe85ef89da4284aad60d902235b57f, State: Running, Role: LEADER
I20260924 16:06:00.886001 12493 consensus_queue.cc:237] T 00000000000000000000000000000000 P bebe85ef89da4284aad60d902235b57f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bebe85ef89da4284aad60d902235b57f" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32885 } }
I20260924 16:06:00.886510 12488 sys_catalog.cc:564] T 00000000000000000000000000000000 P bebe85ef89da4284aad60d902235b57f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:00.887756 12495 sys_catalog.cc:455] T 00000000000000000000000000000000 P bebe85ef89da4284aad60d902235b57f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bebe85ef89da4284aad60d902235b57f. Latest consensus state: current_term: 1 leader_uuid: "bebe85ef89da4284aad60d902235b57f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bebe85ef89da4284aad60d902235b57f" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32885 } } }
I20260924 16:06:00.887897 12495 sys_catalog.cc:458] T 00000000000000000000000000000000 P bebe85ef89da4284aad60d902235b57f [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:00.888231 12494 sys_catalog.cc:455] T 00000000000000000000000000000000 P bebe85ef89da4284aad60d902235b57f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bebe85ef89da4284aad60d902235b57f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bebe85ef89da4284aad60d902235b57f" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32885 } } }
I20260924 16:06:00.888324 12494 sys_catalog.cc:458] T 00000000000000000000000000000000 P bebe85ef89da4284aad60d902235b57f [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:00.888484 12502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:00.891464 12502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:00.897128 12502 catalog_manager.cc:1347] Generated new cluster ID: cd131e57543e450fb44e2ebb3e8515af
I20260924 16:06:00.897199 12502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:00.902684 12502 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:00.903925 12502 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:00.910411 12502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bebe85ef89da4284aad60d902235b57f: Generated new TSK 0
I20260924 16:06:00.911124 12502 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:06:00.913681 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:0
--local_ip_for_outbound_sockets=127.25.109.1
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:32885
--builtin_ntp_servers=127.25.109.20:34489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 16:06:01.026453 12512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:01.026712 12512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:01.026750 12512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:01.030925 12512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:01.030977 12512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:01.031033 12512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:06:01.035917 12512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:32885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:01.037164 12512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:01.038681 12512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:01.048677 12517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:01.048605 12518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:01.048640 12512 server_base.cc:1034] running on GCE node
W20260924 16:06:01.048622 12520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:01.049340 12512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:01.050011 12512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:01.051204 12512 hybrid_clock.cc:648] HybridClock initialized: now 1790265961051171 us; error 45 us; skew 500 ppm
I20260924 16:06:01.053067 12512 webserver.cc:460] Webserver started at http://127.25.109.1:41251/ using document root <none> and password file <none>
I20260924 16:06:01.053704 12512 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:01.053798 12512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:01.054041 12512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:01.056177 12512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-0/data/instance:
uuid: "08c4a237497b40488ad817c90f490a24"
format_stamp: "Formatted at 2026-09-24 16:06:01 on dist-test-slave-n106"
server_key: "c179852faa41922774b05bf9d6233587"
server_key_iv: "f01d8be65fdebce5b964f6b486cbe435"
server_key_version: "encryptionkey@0"
I20260924 16:06:01.056826 12512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance:
uuid: "08c4a237497b40488ad817c90f490a24"
format_stamp: "Formatted at 2026-09-24 16:06:01 on dist-test-slave-n106"
server_key: "c179852faa41922774b05bf9d6233587"
server_key_iv: "f01d8be65fdebce5b964f6b486cbe435"
server_key_version: "encryptionkey@0"
I20260924 16:06:01.060791 12512 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:06:01.063375 12526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:01.064685 12512 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:06:01.064819 12512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "08c4a237497b40488ad817c90f490a24"
format_stamp: "Formatted at 2026-09-24 16:06:01 on dist-test-slave-n106"
server_key: "c179852faa41922774b05bf9d6233587"
server_key_iv: "f01d8be65fdebce5b964f6b486cbe435"
server_key_version: "encryptionkey@0"
I20260924 16:06:01.064939 12512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:01.077556 12512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:01.078315 12512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:01.078526 12512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:01.079149 12512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:06:01.080134 12512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:01.080209 12512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:01.080286 12512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:01.080327 12512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:01.089677 12512 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:45727
I20260924 16:06:01.089699 12639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:45727 every 8 connection(s)
I20260924 16:06:01.090653 12512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:06:01.099781 12491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:01.092597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:39877 (local address 127.25.109.62:32885)
0924 16:06:01.092737 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:01.092747 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:01.093810 (+ 1063us) server_negotiation.cc:407] Connection header received
0924 16:06:01.095280 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:01.095283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:01.095333 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:01.095413 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:01.096324 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.096906 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.098018 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.098247 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.098322 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:01.099366 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:01.099369 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:01.099371 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:01.099383 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:01.099403 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:01.099410 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:01.099466 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:01.099612 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 16:06:01.099664 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 16:06:01.100661 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 12512
I20260924 16:06:01.100788 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance
I20260924 16:06:01.100765 12642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:01.092976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:32885 (local address 127.25.109.1:39877)
0924 16:06:01.093652 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:01.093693 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:01.094974 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:01.095592 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:01.095598 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:01.095686 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:01.096196 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:01.096210 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.097021 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.097023 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:01.097910 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:01.097920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.098381 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.098384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:01.098534 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:01.099117 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:06:01.099119 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:01.099160 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:06:01.099225 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:01.099483 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:01.099490 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:01.099495 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 16:06:01.099592 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 16:06:01.099812 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":119,"threads_started":1}
I20260924 16:06:01.101127 26036 external_mini_cluster.cc:1428] Setting key eb53af05806bb80d5e9a71d3fc091fad
I20260924 16:06:01.102090 12640 heartbeater.cc:347] Connected to a master server at 127.25.109.62:32885
I20260924 16:06:01.102322 12640 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:01.102856 12640 heartbeater.cc:511] Master 127.25.109.62:32885 requested a full tablet report, sending...
I20260924 16:06:01.103286 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:0
--local_ip_for_outbound_sockets=127.25.109.2
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:32885
--builtin_ntp_servers=127.25.109.20:34489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 16:06:01.104573 12452 ts_manager.cc:194] Registered new tserver with Master: 08c4a237497b40488ad817c90f490a24 (127.25.109.1:45727)
I20260924 16:06:01.105826 12452 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.1:39877
W20260924 16:06:01.215118 12643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:01.215564 12643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:01.215654 12643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:01.220566 12643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:01.220626 12643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:01.220713 12643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:06:01.225598 12643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:32885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:01.226742 12643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:01.228168 12643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:01.237973 12651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:01.237881 12649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:01.237932 12643 server_base.cc:1034] running on GCE node
W20260924 16:06:01.237881 12648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:01.238574 12643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:01.239251 12643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:01.240474 12643 hybrid_clock.cc:648] HybridClock initialized: now 1790265961240442 us; error 40 us; skew 500 ppm
I20260924 16:06:01.242192 12643 webserver.cc:460] Webserver started at http://127.25.109.2:33769/ using document root <none> and password file <none>
I20260924 16:06:01.242741 12643 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:01.242801 12643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:01.242964 12643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:01.245114 12643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-1/data/instance:
uuid: "f6feef644fcd417087a1abace2a012fc"
format_stamp: "Formatted at 2026-09-24 16:06:01 on dist-test-slave-n106"
server_key: "14fb1206b88e387d9da69eac38f5a941"
server_key_iv: "779b5b37d42d3225fe363d30bf6f0b48"
server_key_version: "encryptionkey@0"
I20260924 16:06:01.245654 12643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance:
uuid: "f6feef644fcd417087a1abace2a012fc"
format_stamp: "Formatted at 2026-09-24 16:06:01 on dist-test-slave-n106"
server_key: "14fb1206b88e387d9da69eac38f5a941"
server_key_iv: "779b5b37d42d3225fe363d30bf6f0b48"
server_key_version: "encryptionkey@0"
I20260924 16:06:01.249352 12643 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:06:01.251930 12657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:01.253221 12643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:06:01.253350 12643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "f6feef644fcd417087a1abace2a012fc"
format_stamp: "Formatted at 2026-09-24 16:06:01 on dist-test-slave-n106"
server_key: "14fb1206b88e387d9da69eac38f5a941"
server_key_iv: "779b5b37d42d3225fe363d30bf6f0b48"
server_key_version: "encryptionkey@0"
I20260924 16:06:01.253464 12643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:01.270385 12643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:01.271107 12643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:01.271313 12643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:01.271938 12643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:06:01.272986 12643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:01.273034 12643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:01.273108 12643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:01.273145 12643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:01.282276 12643 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:40337
I20260924 16:06:01.282299 12771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:40337 every 8 connection(s)
I20260924 16:06:01.283233 12643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:06:01.290310 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 12643
I20260924 16:06:01.290441 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance
I20260924 16:06:01.290746 26036 external_mini_cluster.cc:1428] Setting key 3ed1382c92a41257b78cb48612df836b
I20260924 16:06:01.292865 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:0
--local_ip_for_outbound_sockets=127.25.109.3
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:32885
--builtin_ntp_servers=127.25.109.20:34489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 16:06:01.293802 12491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:01.285175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:55121 (local address 127.25.109.62:32885)
0924 16:06:01.285314 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:01.285318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:01.286363 (+ 1045us) server_negotiation.cc:407] Connection header received
0924 16:06:01.287986 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:01.287990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:01.288057 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:01.288166 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:01.289385 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.289941 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.291593 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.291890 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.292003 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:01.293281 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:01.293285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:01.293289 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:01.293307 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:01.293326 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:01.293335 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:01.293418 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:01.293584 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 16:06:01.293658 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 16:06:01.294669 12774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:01.285537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:32885 (local address 127.25.109.2:55121)
0924 16:06:01.286206 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:01.286248 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:01.287768 (+ 1520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:01.288369 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:01.288376 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:01.288530 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:01.289239 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:01.289260 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.290496 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.290502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:01.291476 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:01.291487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.292076 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.292081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:01.292269 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:01.292965 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:06:01.292967 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:01.293037 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:06:01.293125 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:01.293423 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:01.293434 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:01.293440 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 16:06:01.293582 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 16:06:01.293882 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":115,"threads_started":1}
I20260924 16:06:01.295805 12772 heartbeater.cc:347] Connected to a master server at 127.25.109.62:32885
I20260924 16:06:01.296059 12772 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:01.296638 12772 heartbeater.cc:511] Master 127.25.109.62:32885 requested a full tablet report, sending...
I20260924 16:06:01.297760 12452 ts_manager.cc:194] Registered new tserver with Master: f6feef644fcd417087a1abace2a012fc (127.25.109.2:40337)
I20260924 16:06:01.298686 12452 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.2:55121
W20260924 16:06:01.407723 12775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:01.408021 12775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:01.408061 12775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:01.412341 12775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:01.412438 12775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:01.412516 12775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:06:01.417677 12775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:32885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:01.418777 12775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:01.420192 12775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:06:01.430267 12775 server_base.cc:1034] running on GCE node
W20260924 16:06:01.430424 12781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:01.430294 12780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:01.430294 12783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:01.431047 12775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:01.431697 12775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:01.432883 12775 hybrid_clock.cc:648] HybridClock initialized: now 1790265961432861 us; error 34 us; skew 500 ppm
I20260924 16:06:01.434809 12775 webserver.cc:460] Webserver started at http://127.25.109.3:46247/ using document root <none> and password file <none>
I20260924 16:06:01.435418 12775 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:01.435509 12775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:01.435742 12775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:01.437973 12775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-2/data/instance:
uuid: "4d38f81a37a44579a8c78d5e9e50e3ca"
format_stamp: "Formatted at 2026-09-24 16:06:01 on dist-test-slave-n106"
server_key: "38c64b5c4728f776f59eadbec19e1b80"
server_key_iv: "2176b84977ed3681ef8aa609ad020558"
server_key_version: "encryptionkey@0"
I20260924 16:06:01.438573 12775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance:
uuid: "4d38f81a37a44579a8c78d5e9e50e3ca"
format_stamp: "Formatted at 2026-09-24 16:06:01 on dist-test-slave-n106"
server_key: "38c64b5c4728f776f59eadbec19e1b80"
server_key_iv: "2176b84977ed3681ef8aa609ad020558"
server_key_version: "encryptionkey@0"
I20260924 16:06:01.442512 12775 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:06:01.445138 12789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:01.446465 12775 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:06:01.446619 12775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "4d38f81a37a44579a8c78d5e9e50e3ca"
format_stamp: "Formatted at 2026-09-24 16:06:01 on dist-test-slave-n106"
server_key: "38c64b5c4728f776f59eadbec19e1b80"
server_key_iv: "2176b84977ed3681ef8aa609ad020558"
server_key_version: "encryptionkey@0"
I20260924 16:06:01.446743 12775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:01.482182 12775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:01.482977 12775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:01.483186 12775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:01.483829 12775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:06:01.484844 12775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:01.484916 12775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:01.485013 12775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:01.485059 12775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:01.494059 12775 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:45755
I20260924 16:06:01.494081 12904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:45755 every 8 connection(s)
I20260924 16:06:01.495029 12775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:06:01.500277 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 12775
I20260924 16:06:01.500423 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance
I20260924 16:06:01.500759 26036 external_mini_cluster.cc:1428] Setting key 12ec61766d02dd5cdfb48794ebb431aa
I20260924 16:06:01.505429 12491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:01.497200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:33367 (local address 127.25.109.62:32885)
0924 16:06:01.497384 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:01.497388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:01.498469 (+ 1081us) server_negotiation.cc:407] Connection header received
0924 16:06:01.499875 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:01.499878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:01.499932 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:01.500011 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:01.501340 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.502098 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.503470 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.503696 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.503783 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:01.504957 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:01.504961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:01.504963 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:01.504976 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:01.504998 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:01.505006 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:01.505073 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:01.505251 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 16:06:01.505309 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 16:06:01.506280 12907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:01.497663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:32885 (local address 127.25.109.3:33367)
0924 16:06:01.498335 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:01.498377 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:01.499657 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:01.500248 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:01.500259 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:01.500388 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:01.501160 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:01.501179 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.502246 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.502249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:01.503338 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:01.503349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.503839 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.503842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:01.503998 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:01.504681 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:06:01.504683 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:01.504726 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:06:01.504797 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:01.505086 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:01.505092 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:01.505097 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 16:06:01.505205 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 16:06:01.505439 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":112,"threads_started":1}
I20260924 16:06:01.507416 12905 heartbeater.cc:347] Connected to a master server at 127.25.109.62:32885
I20260924 16:06:01.507639 12905 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:01.508133 12905 heartbeater.cc:511] Master 127.25.109.62:32885 requested a full tablet report, sending...
I20260924 16:06:01.509215 12452 ts_manager.cc:194] Registered new tserver with Master: 4d38f81a37a44579a8c78d5e9e50e3ca (127.25.109.3:45755)
I20260924 16:06:01.509825 12452 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.3:33367
I20260924 16:06:01.514629 26036 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:06:01.520787 12491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:01.516669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42368 (local address 127.25.109.62:32885)
0924 16:06:01.516835 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:01.516840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:01.517003 (+ 163us) server_negotiation.cc:407] Connection header received
0924 16:06:01.517176 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:01.517180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:01.517233 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:01.517314 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:01.517621 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.518155 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.519029 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.519264 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.519373 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:01.520221 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:01.520225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:01.520227 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:01.520241 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:01.520267 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:01.520275 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:01.520407 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:01.520589 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 16:06:01.520633 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 16:06:01.523787 12452 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:06:01.526950 12452 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:06:01.541751 12917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:01.535730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49890 (local address 127.25.109.2:40337)
0924 16:06:01.536570 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:01.536593 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:01.536623 (+ 30us) server_negotiation.cc:407] Connection header received
0924 16:06:01.536717 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:01.536723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:01.536943 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:01.537129 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:01.537465 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.538721 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.540233 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.541064 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.541185 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:01.541410 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:01.541472 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 16:06:01.541572 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":182,"threads_started":1}
I20260924 16:06:01.541742 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:01.535806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:45727 (local address 127.0.0.1:59216)
0924 16:06:01.537117 (+ 1311us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:01.537131 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:01.537210 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:01.537926 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:01.537929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:01.537949 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:01.538029 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:01.538035 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.539129 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.539132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:01.540288 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:01.540298 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.541335 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.541338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:01.541429 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:01.541456 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 16:06:01.541506 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 16:06:01.541572 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1232,"thread_start_us":110,"threads_started":1}
I20260924 16:06:01.541747 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:01.535537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:40337 (local address 127.0.0.1:49890)
0924 16:06:01.536383 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:01.536459 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:01.536581 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:01.537139 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:01.537145 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:01.537176 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:01.537346 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:01.537352 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.538851 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.538855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:01.540095 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:01.540106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.541192 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.541195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:01.541299 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:01.541340 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 16:06:01.541443 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 16:06:01.541562 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":105,"threads_started":1}
I20260924 16:06:01.542409 12919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:01.536079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59216 (local address 127.25.109.1:45727)
0924 16:06:01.537466 (+ 1387us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:01.537478 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:01.537504 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:06:01.537598 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:01.537603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:01.537793 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:01.537950 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:01.538132 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.539027 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.540441 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.541201 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.541952 (+ 751us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:01.542112 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:01.542152 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 16:06:01.542219 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1268,"thread_start_us":114,"threads_started":1}
I20260924 16:06:01.542656 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:01.536726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60492 (local address 127.25.109.3:45755)
0924 16:06:01.537101 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:01.537113 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:01.537600 (+ 487us) server_negotiation.cc:407] Connection header received
0924 16:06:01.537674 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:01.537678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:01.537822 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:01.537950 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:01.538353 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.539676 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.540877 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.541589 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.541699 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:01.542303 (+ 604us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:01.542354 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 16:06:01.542465 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":117,"threads_started":1}
I20260924 16:06:01.543108 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:01.536289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:45755 (local address 127.0.0.1:60492)
0924 16:06:01.537385 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:01.537396 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:01.537479 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:01.538114 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:01.538118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:01.538137 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:01.538225 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:01.538230 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.539802 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.539806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:01.540762 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:01.540770 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.541978 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.541984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:01.542079 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:01.542106 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 16:06:01.542167 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 16:06:01.542228 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1038,"thread_start_us":83,"threads_started":1}
I20260924 16:06:01.545349 12574 tablet_service.cc:1467] Processing CreateTablet for tablet 10101fe2ed434bb4a4795a034fd63ee1 (DEFAULT_TABLE table=test-table [id=10cac218c1c64d33a5c7590a3d6fd22a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:01.545506 12705 tablet_service.cc:1467] Processing CreateTablet for tablet 10101fe2ed434bb4a4795a034fd63ee1 (DEFAULT_TABLE table=test-table [id=10cac218c1c64d33a5c7590a3d6fd22a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:01.545508 12838 tablet_service.cc:1467] Processing CreateTablet for tablet 10101fe2ed434bb4a4795a034fd63ee1 (DEFAULT_TABLE table=test-table [id=10cac218c1c64d33a5c7590a3d6fd22a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:01.546422 12574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10101fe2ed434bb4a4795a034fd63ee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:01.546533 12838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10101fe2ed434bb4a4795a034fd63ee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:01.546533 12705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10101fe2ed434bb4a4795a034fd63ee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:01.553026 12922 tablet_bootstrap.cc:492] T 10101fe2ed434bb4a4795a034fd63ee1 P 08c4a237497b40488ad817c90f490a24: Bootstrap starting.
I20260924 16:06:01.554463 12924 tablet_bootstrap.cc:492] T 10101fe2ed434bb4a4795a034fd63ee1 P 4d38f81a37a44579a8c78d5e9e50e3ca: Bootstrap starting.
I20260924 16:06:01.554548 12923 tablet_bootstrap.cc:492] T 10101fe2ed434bb4a4795a034fd63ee1 P f6feef644fcd417087a1abace2a012fc: Bootstrap starting.
I20260924 16:06:01.555572 12922 tablet_bootstrap.cc:654] T 10101fe2ed434bb4a4795a034fd63ee1 P 08c4a237497b40488ad817c90f490a24: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:01.556301 12924 tablet_bootstrap.cc:654] T 10101fe2ed434bb4a4795a034fd63ee1 P 4d38f81a37a44579a8c78d5e9e50e3ca: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:01.556506 12922 log.cc:824] T 10101fe2ed434bb4a4795a034fd63ee1 P 08c4a237497b40488ad817c90f490a24: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:01.556644 12923 tablet_bootstrap.cc:654] T 10101fe2ed434bb4a4795a034fd63ee1 P f6feef644fcd417087a1abace2a012fc: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:01.557219 12924 log.cc:824] T 10101fe2ed434bb4a4795a034fd63ee1 P 4d38f81a37a44579a8c78d5e9e50e3ca: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:01.557469 12923 log.cc:824] T 10101fe2ed434bb4a4795a034fd63ee1 P f6feef644fcd417087a1abace2a012fc: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:01.559098 12922 tablet_bootstrap.cc:492] T 10101fe2ed434bb4a4795a034fd63ee1 P 08c4a237497b40488ad817c90f490a24: No bootstrap required, opened a new log
I20260924 16:06:01.559221 12924 tablet_bootstrap.cc:492] T 10101fe2ed434bb4a4795a034fd63ee1 P 4d38f81a37a44579a8c78d5e9e50e3ca: No bootstrap required, opened a new log
I20260924 16:06:01.559314 12922 ts_tablet_manager.cc:1397] T 10101fe2ed434bb4a4795a034fd63ee1 P 08c4a237497b40488ad817c90f490a24: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 16:06:01.559381 12923 tablet_bootstrap.cc:492] T 10101fe2ed434bb4a4795a034fd63ee1 P f6feef644fcd417087a1abace2a012fc: No bootstrap required, opened a new log
I20260924 16:06:01.559456 12924 ts_tablet_manager.cc:1397] T 10101fe2ed434bb4a4795a034fd63ee1 P 4d38f81a37a44579a8c78d5e9e50e3ca: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 16:06:01.559551 12923 ts_tablet_manager.cc:1397] T 10101fe2ed434bb4a4795a034fd63ee1 P f6feef644fcd417087a1abace2a012fc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 16:06:01.562767 12924 raft_consensus.cc:348] T 10101fe2ed434bb4a4795a034fd63ee1 P 4d38f81a37a44579a8c78d5e9e50e3ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6feef644fcd417087a1abace2a012fc" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 40337 } } peers { permanent_uuid: "08c4a237497b40488ad817c90f490a24" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 45727 } } peers { permanent_uuid: "4d38f81a37a44579a8c78d5e9e50e3ca" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 45755 } }
I20260924 16:06:01.563014 12924 raft_consensus.cc:374] T 10101fe2ed434bb4a4795a034fd63ee1 P 4d38f81a37a44579a8c78d5e9e50e3ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:01.563067 12924 raft_consensus.cc:729] T 10101fe2ed434bb4a4795a034fd63ee1 P 4d38f81a37a44579a8c78d5e9e50e3ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d38f81a37a44579a8c78d5e9e50e3ca, State: Initialized, Role: FOLLOWER
I20260924 16:06:01.563534 12924 consensus_queue.cc:260] T 10101fe2ed434bb4a4795a034fd63ee1 P 4d38f81a37a44579a8c78d5e9e50e3ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6feef644fcd417087a1abace2a012fc" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 40337 } } peers { permanent_uuid: "08c4a237497b40488ad817c90f490a24" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 45727 } } peers { permanent_uuid: "4d38f81a37a44579a8c78d5e9e50e3ca" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 45755 } }
I20260924 16:06:01.563676 12922 raft_consensus.cc:348] T 10101fe2ed434bb4a4795a034fd63ee1 P 08c4a237497b40488ad817c90f490a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6feef644fcd417087a1abace2a012fc" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 40337 } } peers { permanent_uuid: "08c4a237497b40488ad817c90f490a24" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 45727 } } peers { permanent_uuid: "4d38f81a37a44579a8c78d5e9e50e3ca" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 45755 } }
I20260924 16:06:01.563747 12923 raft_consensus.cc:348] T 10101fe2ed434bb4a4795a034fd63ee1 P f6feef644fcd417087a1abace2a012fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6feef644fcd417087a1abace2a012fc" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 40337 } } peers { permanent_uuid: "08c4a237497b40488ad817c90f490a24" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 45727 } } peers { permanent_uuid: "4d38f81a37a44579a8c78d5e9e50e3ca" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 45755 } }
I20260924 16:06:01.563946 12923 raft_consensus.cc:374] T 10101fe2ed434bb4a4795a034fd63ee1 P f6feef644fcd417087a1abace2a012fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:01.563946 12922 raft_consensus.cc:374] T 10101fe2ed434bb4a4795a034fd63ee1 P 08c4a237497b40488ad817c90f490a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:01.564007 12922 raft_consensus.cc:729] T 10101fe2ed434bb4a4795a034fd63ee1 P 08c4a237497b40488ad817c90f490a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08c4a237497b40488ad817c90f490a24, State: Initialized, Role: FOLLOWER
I20260924 16:06:01.564427 12905 heartbeater.cc:503] Master 127.25.109.62:32885 was elected leader, sending a full tablet report...
I20260924 16:06:01.564007 12923 raft_consensus.cc:729] T 10101fe2ed434bb4a4795a034fd63ee1 P f6feef644fcd417087a1abace2a012fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6feef644fcd417087a1abace2a012fc, State: Initialized, Role: FOLLOWER
I20260924 16:06:01.564678 12924 ts_tablet_manager.cc:1428] T 10101fe2ed434bb4a4795a034fd63ee1 P 4d38f81a37a44579a8c78d5e9e50e3ca: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 16:06:01.564977 12923 consensus_queue.cc:260] T 10101fe2ed434bb4a4795a034fd63ee1 P f6feef644fcd417087a1abace2a012fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6feef644fcd417087a1abace2a012fc" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 40337 } } peers { permanent_uuid: "08c4a237497b40488ad817c90f490a24" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 45727 } } peers { permanent_uuid: "4d38f81a37a44579a8c78d5e9e50e3ca" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 45755 } }
I20260924 16:06:01.565629 12772 heartbeater.cc:503] Master 127.25.109.62:32885 was elected leader, sending a full tablet report...
I20260924 16:06:01.565773 12922 consensus_queue.cc:260] T 10101fe2ed434bb4a4795a034fd63ee1 P 08c4a237497b40488ad817c90f490a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6feef644fcd417087a1abace2a012fc" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 40337 } } peers { permanent_uuid: "08c4a237497b40488ad817c90f490a24" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 45727 } } peers { permanent_uuid: "4d38f81a37a44579a8c78d5e9e50e3ca" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 45755 } }
I20260924 16:06:01.565909 12923 ts_tablet_manager.cc:1428] T 10101fe2ed434bb4a4795a034fd63ee1 P f6feef644fcd417087a1abace2a012fc: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 16:06:01.566579 12922 ts_tablet_manager.cc:1428] T 10101fe2ed434bb4a4795a034fd63ee1 P 08c4a237497b40488ad817c90f490a24: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 16:06:01.566897 12640 heartbeater.cc:503] Master 127.25.109.62:32885 was elected leader, sending a full tablet report...
W20260924 16:06:01.591929 12641 tablet.cc:2369] T 10101fe2ed434bb4a4795a034fd63ee1 P 08c4a237497b40488ad817c90f490a24: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:06:01.746066 12906 tablet.cc:2369] T 10101fe2ed434bb4a4795a034fd63ee1 P 4d38f81a37a44579a8c78d5e9e50e3ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:06:01.784554 12773 tablet.cc:2369] T 10101fe2ed434bb4a4795a034fd63ee1 P f6feef644fcd417087a1abace2a012fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:06:01.954599 12930 raft_consensus.cc:482] T 10101fe2ed434bb4a4795a034fd63ee1 P 08c4a237497b40488ad817c90f490a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:06:01.954825 12930 raft_consensus.cc:504] T 10101fe2ed434bb4a4795a034fd63ee1 P 08c4a237497b40488ad817c90f490a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6feef644fcd417087a1abace2a012fc" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 40337 } } peers { permanent_uuid: "08c4a237497b40488ad817c90f490a24" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 45727 } } peers { permanent_uuid: "4d38f81a37a44579a8c78d5e9e50e3ca" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 45755 } }
I20260924 16:06:01.956089 12930 leader_election.cc:288] T 10101fe2ed434bb4a4795a034fd63ee1 P 08c4a237497b40488ad817c90f490a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6feef644fcd417087a1abace2a012fc (127.25.109.2:40337), 4d38f81a37a44579a8c78d5e9e50e3ca (127.25.109.3:45755)
I20260924 16:06:01.960273 12917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:01.956440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:45805 (local address 127.25.109.2:40337)
0924 16:06:01.956608 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:01.956615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:01.956730 (+ 115us) server_negotiation.cc:407] Connection header received
0924 16:06:01.956970 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:01.956974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:01.957031 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:01.957115 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:01.957446 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.958100 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.959256 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.959665 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.959805 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:01.959859 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:01.960046 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 16:06:01.960133 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 16:06:01.960285 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:01.956696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:44381 (local address 127.25.109.3:45755)
0924 16:06:01.956817 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:01.956821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:01.956972 (+ 151us) server_negotiation.cc:407] Connection header received
0924 16:06:01.957122 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:01.957125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:01.957177 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:01.957258 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:01.957610 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.958206 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.959289 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.959670 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.959794 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:01.959848 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:01.960096 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 16:06:01.960180 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 16:06:01.960294 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:01.956317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:40337 (local address 127.25.109.1:45805)
0924 16:06:01.956636 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:01.956651 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:01.956821 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:01.957137 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:01.957140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:01.957157 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:01.957315 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:01.957324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.958211 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.958214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:01.959121 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:01.959131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.959794 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.959798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:01.959918 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:01.959938 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 16:06:01.959999 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 16:06:01.960084 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":5,"thread_start_us":95,"threads_started":1}
I20260924 16:06:01.960294 12932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:01.956437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:45755 (local address 127.25.109.1:44381)
0924 16:06:01.956850 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:01.956862 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:01.956995 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:01.957293 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:01.957295 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:01.957311 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:01.957461 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:01.957467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.958321 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.958323 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:01.959131 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:01.959141 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.959806 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.959810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:01.959950 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:01.959968 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 16:06:01.960066 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 16:06:01.960138 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":25,"thread_start_us":152,"threads_started":1}
I20260924 16:06:01.961129 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10101fe2ed434bb4a4795a034fd63ee1" candidate_uuid: "08c4a237497b40488ad817c90f490a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6feef644fcd417087a1abace2a012fc" is_pre_election: true
I20260924 16:06:01.961136 12858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10101fe2ed434bb4a4795a034fd63ee1" candidate_uuid: "08c4a237497b40488ad817c90f490a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d38f81a37a44579a8c78d5e9e50e3ca" is_pre_election: true
I20260924 16:06:01.961447 12858 raft_consensus.cc:2454] T 10101fe2ed434bb4a4795a034fd63ee1 P 4d38f81a37a44579a8c78d5e9e50e3ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08c4a237497b40488ad817c90f490a24 in term 0.
I20260924 16:06:01.961460 12726 raft_consensus.cc:2454] T 10101fe2ed434bb4a4795a034fd63ee1 P f6feef644fcd417087a1abace2a012fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08c4a237497b40488ad817c90f490a24 in term 0.
I20260924 16:06:01.962007 12528 leader_election.cc:302] T 10101fe2ed434bb4a4795a034fd63ee1 P 08c4a237497b40488ad817c90f490a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 08c4a237497b40488ad817c90f490a24, 4d38f81a37a44579a8c78d5e9e50e3ca, f6feef644fcd417087a1abace2a012fc; no voters:
I20260924 16:06:01.962301 12930 raft_consensus.cc:2781] T 10101fe2ed434bb4a4795a034fd63ee1 P 08c4a237497b40488ad817c90f490a24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:06:01.962388 12930 raft_consensus.cc:482] T 10101fe2ed434bb4a4795a034fd63ee1 P 08c4a237497b40488ad817c90f490a24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:06:01.962441 12930 raft_consensus.cc:3037] T 10101fe2ed434bb4a4795a034fd63ee1 P 08c4a237497b40488ad817c90f490a24 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:01.963531 12930 raft_consensus.cc:504] T 10101fe2ed434bb4a4795a034fd63ee1 P 08c4a237497b40488ad817c90f490a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6feef644fcd417087a1abace2a012fc" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 40337 } } peers { permanent_uuid: "08c4a237497b40488ad817c90f490a24" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 45727 } } peers { permanent_uuid: "4d38f81a37a44579a8c78d5e9e50e3ca" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 45755 } }
I20260924 16:06:01.963900 12930 leader_election.cc:288] T 10101fe2ed434bb4a4795a034fd63ee1 P 08c4a237497b40488ad817c90f490a24 [CANDIDATE]: Term 1 election: Requested vote from peers f6feef644fcd417087a1abace2a012fc (127.25.109.2:40337), 4d38f81a37a44579a8c78d5e9e50e3ca (127.25.109.3:45755)
I20260924 16:06:01.964249 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10101fe2ed434bb4a4795a034fd63ee1" candidate_uuid: "08c4a237497b40488ad817c90f490a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6feef644fcd417087a1abace2a012fc"
I20260924 16:06:01.964322 12858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10101fe2ed434bb4a4795a034fd63ee1" candidate_uuid: "08c4a237497b40488ad817c90f490a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d38f81a37a44579a8c78d5e9e50e3ca"
I20260924 16:06:01.964390 12726 raft_consensus.cc:3037] T 10101fe2ed434bb4a4795a034fd63ee1 P f6feef644fcd417087a1abace2a012fc [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:01.964454 12858 raft_consensus.cc:3037] T 10101fe2ed434bb4a4795a034fd63ee1 P 4d38f81a37a44579a8c78d5e9e50e3ca [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:01.965781 12726 raft_consensus.cc:2454] T 10101fe2ed434bb4a4795a034fd63ee1 P f6feef644fcd417087a1abace2a012fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08c4a237497b40488ad817c90f490a24 in term 1.
I20260924 16:06:01.965781 12858 raft_consensus.cc:2454] T 10101fe2ed434bb4a4795a034fd63ee1 P 4d38f81a37a44579a8c78d5e9e50e3ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08c4a237497b40488ad817c90f490a24 in term 1.
I20260924 16:06:01.966168 12528 leader_election.cc:302] T 10101fe2ed434bb4a4795a034fd63ee1 P 08c4a237497b40488ad817c90f490a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08c4a237497b40488ad817c90f490a24, f6feef644fcd417087a1abace2a012fc; no voters:
I20260924 16:06:01.966405 12930 raft_consensus.cc:2781] T 10101fe2ed434bb4a4795a034fd63ee1 P 08c4a237497b40488ad817c90f490a24 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:01.966660 12930 raft_consensus.cc:686] T 10101fe2ed434bb4a4795a034fd63ee1 P 08c4a237497b40488ad817c90f490a24 [term 1 LEADER]: Becoming Leader. State: Replica: 08c4a237497b40488ad817c90f490a24, State: Running, Role: LEADER
I20260924 16:06:01.967036 12930 consensus_queue.cc:237] T 10101fe2ed434bb4a4795a034fd63ee1 P 08c4a237497b40488ad817c90f490a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6feef644fcd417087a1abace2a012fc" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 40337 } } peers { permanent_uuid: "08c4a237497b40488ad817c90f490a24" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 45727 } } peers { permanent_uuid: "4d38f81a37a44579a8c78d5e9e50e3ca" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 45755 } }
I20260924 16:06:01.970839 12450 catalog_manager.cc:5495] T 10101fe2ed434bb4a4795a034fd63ee1 P 08c4a237497b40488ad817c90f490a24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08c4a237497b40488ad817c90f490a24 (127.25.109.1). New cstate: current_term: 1 leader_uuid: "08c4a237497b40488ad817c90f490a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6feef644fcd417087a1abace2a012fc" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 40337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08c4a237497b40488ad817c90f490a24" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 45727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d38f81a37a44579a8c78d5e9e50e3ca" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 45755 } health_report { overall_health: UNKNOWN } } }
I20260924 16:06:01.998371 12919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:01.994810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59226 (local address 127.25.109.1:45727)
0924 16:06:01.994965 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:01.994969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:01.994982 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:06:01.995059 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:01.995061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:01.995121 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:01.995214 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:06:01.995454 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.995968 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.996934 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.997152 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.997232 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:01.997543 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:06:01.997989 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:06:01.998105 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:01.998218 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 16:06:01.998254 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:06:02.004920 12726 raft_consensus.cc:1261] T 10101fe2ed434bb4a4795a034fd63ee1 P f6feef644fcd417087a1abace2a012fc [term 1 FOLLOWER]: Refusing update from remote peer 08c4a237497b40488ad817c90f490a24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:06:02.004920 12858 raft_consensus.cc:1261] T 10101fe2ed434bb4a4795a034fd63ee1 P 4d38f81a37a44579a8c78d5e9e50e3ca [term 1 FOLLOWER]: Refusing update from remote peer 08c4a237497b40488ad817c90f490a24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:06:02.005679 12930 consensus_queue.cc:1035] T 10101fe2ed434bb4a4795a034fd63ee1 P 08c4a237497b40488ad817c90f490a24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6feef644fcd417087a1abace2a012fc" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 40337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:06:02.005942 12933 consensus_queue.cc:1035] T 10101fe2ed434bb4a4795a034fd63ee1 P 08c4a237497b40488ad817c90f490a24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d38f81a37a44579a8c78d5e9e50e3ca" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 45755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:06:02.022522 12450 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:06:02.022720 12450 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:06:02.025452 12450 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bebe85ef89da4284aad60d902235b57f: Sending DeleteTablet for 3 replicas of tablet 10101fe2ed434bb4a4795a034fd63ee1
I20260924 16:06:02.026299 12705 tablet_service.cc:1514] Processing DeleteTablet for tablet 10101fe2ed434bb4a4795a034fd63ee1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:02 UTC) from {username='slave'} at 127.0.0.1:49890
I20260924 16:06:02.026522 12574 tablet_service.cc:1514] Processing DeleteTablet for tablet 10101fe2ed434bb4a4795a034fd63ee1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:02 UTC) from {username='slave'} at 127.0.0.1:59216
I20260924 16:06:02.027047 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 12512
I20260924 16:06:02.026984 12838 tablet_service.cc:1514] Processing DeleteTablet for tablet 10101fe2ed434bb4a4795a034fd63ee1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:02 UTC) from {username='slave'} at 127.0.0.1:60492
I20260924 16:06:02.028239 12947 tablet_replica.cc:330] T 10101fe2ed434bb4a4795a034fd63ee1 P 4d38f81a37a44579a8c78d5e9e50e3ca: stopping tablet replica
I20260924 16:06:02.028668 12947 raft_consensus.cc:2229] T 10101fe2ed434bb4a4795a034fd63ee1 P 4d38f81a37a44579a8c78d5e9e50e3ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:06:02.029037 12947 raft_consensus.cc:2258] T 10101fe2ed434bb4a4795a034fd63ee1 P 4d38f81a37a44579a8c78d5e9e50e3ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:02.029834 12945 tablet_replica.cc:330] T 10101fe2ed434bb4a4795a034fd63ee1 P f6feef644fcd417087a1abace2a012fc: stopping tablet replica
I20260924 16:06:02.030149 12945 raft_consensus.cc:2229] T 10101fe2ed434bb4a4795a034fd63ee1 P f6feef644fcd417087a1abace2a012fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:06:02.030453 12945 raft_consensus.cc:2258] T 10101fe2ed434bb4a4795a034fd63ee1 P f6feef644fcd417087a1abace2a012fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:02.030606 12947 ts_tablet_manager.cc:1905] T 10101fe2ed434bb4a4795a034fd63ee1 P 4d38f81a37a44579a8c78d5e9e50e3ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:02.031888 12945 ts_tablet_manager.cc:1905] T 10101fe2ed434bb4a4795a034fd63ee1 P f6feef644fcd417087a1abace2a012fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:02.033612 12947 ts_tablet_manager.cc:1918] T 10101fe2ed434bb4a4795a034fd63ee1 P 4d38f81a37a44579a8c78d5e9e50e3ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:06:02.033725 12947 log.cc:1196] T 10101fe2ed434bb4a4795a034fd63ee1 P 4d38f81a37a44579a8c78d5e9e50e3ca: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-2/wal/wals/10101fe2ed434bb4a4795a034fd63ee1
I20260924 16:06:02.034060 12947 ts_tablet_manager.cc:1939] T 10101fe2ed434bb4a4795a034fd63ee1 P 4d38f81a37a44579a8c78d5e9e50e3ca: Deleting consensus metadata
I20260924 16:06:02.034796 12945 ts_tablet_manager.cc:1918] T 10101fe2ed434bb4a4795a034fd63ee1 P f6feef644fcd417087a1abace2a012fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:06:02.034901 12945 log.cc:1196] T 10101fe2ed434bb4a4795a034fd63ee1 P f6feef644fcd417087a1abace2a012fc: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265708917108-26036-0/minicluster-data/ts-1/wal/wals/10101fe2ed434bb4a4795a034fd63ee1
I20260924 16:06:02.035254 12945 ts_tablet_manager.cc:1939] T 10101fe2ed434bb4a4795a034fd63ee1 P f6feef644fcd417087a1abace2a012fc: Deleting consensus metadata
I20260924 16:06:02.035547 12439 catalog_manager.cc:4841] TS 4d38f81a37a44579a8c78d5e9e50e3ca (127.25.109.3:45755): tablet 10101fe2ed434bb4a4795a034fd63ee1 (table test-table [id=10cac218c1c64d33a5c7590a3d6fd22a]) successfully deleted
W20260924 16:06:02.035892 12439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.109.1:45727 (error 108)
I20260924 16:06:02.036163 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 12643
I20260924 16:06:02.037262 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:02.036945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:45727 (local address 127.0.0.1:59242)
0924 16:06:02.037098 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:02.037174 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.1:45727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 16:06:02.037423 12439 catalog_manager.cc:4584] TS 08c4a237497b40488ad817c90f490a24 (127.25.109.1:45727): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 10101fe2ed434bb4a4795a034fd63ee1: Network error: Client connection negotiation failed: client connection to 127.25.109.1:45727: connect: Connection refused (error 111)
I20260924 16:06:02.043499 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 12775
I20260924 16:06:02.044713 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:02.043683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:40337 (local address 127.0.0.1:49900)
0924 16:06:02.044525 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:02.044601 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.2:40337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":697}
I20260924 16:06:02.050053 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 12421
2026-09-24T16:06:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1395 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:06:02 dist-test-slave-n106 krb5kdc[12951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:06:02 dist-test-slave-n106 krb5kdc[12951](info): set up 2 sockets
Sep 24 16:06:02 dist-test-slave-n106 krb5kdc[12951](info): commencing operation
krb5kdc: starting...
W20260924 16:06:03.089438 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:06:03 dist-test-slave-n106 krb5kdc[12951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:06:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
I20260924 16:06:03.215975 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:40899
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:40641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:40899
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:03.332629 12965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:03.332909 12965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:03.333012 12965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:03.337062 12965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:03.337121 12965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:03.337141 12965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:03.337157 12965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:03.337212 12965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:03.342643 12965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:40641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:40899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:40899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:03.343827 12965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:03.344969 12965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:03.353870 12971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:03.354009 12965 server_base.cc:1034] running on GCE node
W20260924 16:06:03.353928 12973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:03.353888 12970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:03.354714 12965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:03.355866 12965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:03.357085 12965 hybrid_clock.cc:648] HybridClock initialized: now 1790265963357068 us; error 30 us; skew 500 ppm
Sep 24 16:06:03 dist-test-slave-n106 krb5kdc[12951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265963, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:03.360152 12965 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:06:03.361160 12965 webserver.cc:460] Webserver started at http://127.25.109.62:44643/ using document root <none> and password file <none>
I20260924 16:06:03.361769 12965 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:03.361864 12965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:03.362107 12965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:03.364302 12965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "2da08b3aa11b478e92c25a790092ed02"
format_stamp: "Formatted at 2026-09-24 16:06:03 on dist-test-slave-n106"
server_key: "7067fa60d1022b46672d1bceb9b9980d"
server_key_iv: "7b51e4ecfd6dcdfcb1972adfb37a67fe"
server_key_version: "encryptionkey@0"
I20260924 16:06:03.364966 12965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "2da08b3aa11b478e92c25a790092ed02"
format_stamp: "Formatted at 2026-09-24 16:06:03 on dist-test-slave-n106"
server_key: "7067fa60d1022b46672d1bceb9b9980d"
server_key_iv: "7b51e4ecfd6dcdfcb1972adfb37a67fe"
server_key_version: "encryptionkey@0"
I20260924 16:06:03.368885 12965 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:06:03.371376 12980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:03.372735 12965 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:06:03.372874 12965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "2da08b3aa11b478e92c25a790092ed02"
format_stamp: "Formatted at 2026-09-24 16:06:03 on dist-test-slave-n106"
server_key: "7067fa60d1022b46672d1bceb9b9980d"
server_key_iv: "7b51e4ecfd6dcdfcb1972adfb37a67fe"
server_key_version: "encryptionkey@0"
I20260924 16:06:03.372994 12965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:03.395275 12965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:03.398902 12965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:03.399150 12965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:03.406509 12965 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:40899
I20260924 16:06:03.406535 13032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:40899 every 8 connection(s)
I20260924 16:06:03.407475 12965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:03.410419 13033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:03.413224 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 12965
I20260924 16:06:03.413400 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:06:03.413717 26036 external_mini_cluster.cc:1428] Setting key 5a4dd04afb28016c4d0731e49393b227
I20260924 16:06:03.416951 13033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2da08b3aa11b478e92c25a790092ed02: Bootstrap starting.
I20260924 16:06:03.419688 13033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2da08b3aa11b478e92c25a790092ed02: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:03.420645 13033 log.cc:824] T 00000000000000000000000000000000 P 2da08b3aa11b478e92c25a790092ed02: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:03.422631 13033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2da08b3aa11b478e92c25a790092ed02: No bootstrap required, opened a new log
Sep 24 16:06:03 dist-test-slave-n106 krb5kdc[12951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:06:03.427562 13033 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2da08b3aa11b478e92c25a790092ed02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da08b3aa11b478e92c25a790092ed02" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40899 } }
I20260924 16:06:03.427902 13033 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2da08b3aa11b478e92c25a790092ed02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:03.427956 13033 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2da08b3aa11b478e92c25a790092ed02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2da08b3aa11b478e92c25a790092ed02, State: Initialized, Role: FOLLOWER
I20260924 16:06:03.428489 13033 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2da08b3aa11b478e92c25a790092ed02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da08b3aa11b478e92c25a790092ed02" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40899 } }
I20260924 16:06:03.428671 13033 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2da08b3aa11b478e92c25a790092ed02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:03.428740 13033 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2da08b3aa11b478e92c25a790092ed02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:03.428835 13033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2da08b3aa11b478e92c25a790092ed02 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:03.429262 13036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:03.415256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35824 (local address 127.25.109.62:40899)
0924 16:06:03.415735 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:03.415751 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:03.415788 (+ 37us) server_negotiation.cc:407] Connection header received
0924 16:06:03.416464 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:03.416493 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:03.416850 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:03.417190 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:03.418083 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:03.419081 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:03.420313 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:03.420908 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:03.421062 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:03.423694 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:03.423701 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:03.423719 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:03.423751 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:03.425927 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:03.426399 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:03.426405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:03.426412 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:03.426557 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:03.426836 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:03.426839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:03.426841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:03.427372 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:03.427571 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:03.427846 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 16:06:03.428158 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":146,"threads_started":1}
I20260924 16:06:03.430058 13033 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2da08b3aa11b478e92c25a790092ed02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da08b3aa11b478e92c25a790092ed02" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40899 } }
I20260924 16:06:03.430431 13033 leader_election.cc:302] T 00000000000000000000000000000000 P 2da08b3aa11b478e92c25a790092ed02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2da08b3aa11b478e92c25a790092ed02; no voters:
I20260924 16:06:03.430850 13033 leader_election.cc:288] T 00000000000000000000000000000000 P 2da08b3aa11b478e92c25a790092ed02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:06:03.430967 13038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2da08b3aa11b478e92c25a790092ed02 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:03.431181 13038 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2da08b3aa11b478e92c25a790092ed02 [term 1 LEADER]: Becoming Leader. State: Replica: 2da08b3aa11b478e92c25a790092ed02, State: Running, Role: LEADER
I20260924 16:06:03.431514 13038 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2da08b3aa11b478e92c25a790092ed02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da08b3aa11b478e92c25a790092ed02" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40899 } }
I20260924 16:06:03.432267 13033 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2da08b3aa11b478e92c25a790092ed02 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:03.433504 13040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2da08b3aa11b478e92c25a790092ed02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2da08b3aa11b478e92c25a790092ed02. Latest consensus state: current_term: 1 leader_uuid: "2da08b3aa11b478e92c25a790092ed02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da08b3aa11b478e92c25a790092ed02" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40899 } } }
I20260924 16:06:03.433650 13040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2da08b3aa11b478e92c25a790092ed02 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:03.433506 13039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2da08b3aa11b478e92c25a790092ed02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2da08b3aa11b478e92c25a790092ed02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da08b3aa11b478e92c25a790092ed02" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 40899 } } }
I20260924 16:06:03.433844 13039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2da08b3aa11b478e92c25a790092ed02 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:03.434110 13047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:03.437484 13047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:03.443508 13047 catalog_manager.cc:1347] Generated new cluster ID: 328044dfc8a848e98cbf0f537a76d734
I20260924 16:06:03.443591 13047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:03.494572 13047 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:03.496012 13047 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:03.512547 13047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2da08b3aa11b478e92c25a790092ed02: Generated new TSK 0
I20260924 16:06:03.513345 13047 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
Entry for principal kudu/127.25.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab.
I20260924 16:06:03.551813 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:0
--local_ip_for_outbound_sockets=127.25.109.1
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:40899
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:40641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:03.668124 13059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:03.668443 13059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:03.668490 13059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:03.672708 13059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:03.672807 13059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:03.672917 13059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:06:03.678857 13059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:40641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.1.keytab
--principal=kudu/127.25.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:40899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:03.679966 13059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:03.681022 13059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:03.690954 13065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:03.690970 13064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:03.690961 13067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:03.691423 13059 server_base.cc:1034] running on GCE node
I20260924 16:06:03.691814 13059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:03.692483 13059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:03.693661 13059 hybrid_clock.cc:648] HybridClock initialized: now 1790265963693643 us; error 52 us; skew 500 ppm
Sep 24 16:06:03 dist-test-slave-n106 krb5kdc[12951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265963, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:03.696676 13059 init.cc:377] Logged in from keytab as kudu/127.25.109.1@KRBTEST.COM (short username kudu)
I20260924 16:06:03.697659 13059 webserver.cc:460] Webserver started at http://127.25.109.1:38435/ using document root <none> and password file <none>
I20260924 16:06:03.698277 13059 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:03.698374 13059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:03.698621 13059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:03.701362 13059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-0/data/instance:
uuid: "9506d35a969143d7bb420782966b9a38"
format_stamp: "Formatted at 2026-09-24 16:06:03 on dist-test-slave-n106"
server_key: "2bc46a87a5247fa78afc73a3488dfb3c"
server_key_iv: "05814d2a592ac3905d07a7defb8a2782"
server_key_version: "encryptionkey@0"
I20260924 16:06:03.701975 13059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance:
uuid: "9506d35a969143d7bb420782966b9a38"
format_stamp: "Formatted at 2026-09-24 16:06:03 on dist-test-slave-n106"
server_key: "2bc46a87a5247fa78afc73a3488dfb3c"
server_key_iv: "05814d2a592ac3905d07a7defb8a2782"
server_key_version: "encryptionkey@0"
I20260924 16:06:03.705952 13059 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:06:03.708664 13074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:03.709931 13059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:06:03.710088 13059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "9506d35a969143d7bb420782966b9a38"
format_stamp: "Formatted at 2026-09-24 16:06:03 on dist-test-slave-n106"
server_key: "2bc46a87a5247fa78afc73a3488dfb3c"
server_key_iv: "05814d2a592ac3905d07a7defb8a2782"
server_key_version: "encryptionkey@0"
I20260924 16:06:03.710217 13059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:03.748525 13059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:03.751813 13059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:03.752039 13059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:03.752724 13059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:06:03.753609 13059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:03.753691 13059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:03.753762 13059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:03.753814 13059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:03.762780 13059 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:43483
I20260924 16:06:03.762804 13188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:43483 every 8 connection(s)
I20260924 16:06:03.763751 13059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:06:03.769176 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 13059
I20260924 16:06:03.769321 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance
I20260924 16:06:03.769622 26036 external_mini_cluster.cc:1428] Setting key 01ee40ad8f0e558da0d6598962a7d116
Sep 24 16:06:03 dist-test-slave-n106 krb5kdc[12951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265963, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.1@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:06:03.778792 13036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:03.765714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:47515 (local address 127.25.109.62:40899)
0924 16:06:03.765854 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:03.765857 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:03.766877 (+ 1020us) server_negotiation.cc:407] Connection header received
0924 16:06:03.767739 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:03.767742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:03.767792 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:03.767867 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:03.769109 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:03.769809 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:03.771233 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:03.771500 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:03.771591 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:03.774555 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:03.774562 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:03.774569 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:03.774605 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:03.776354 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:03.776936 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:03.776943 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:03.776948 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:03.777051 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:03.777472 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:03.777477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:03.777481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:03.777774 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:03.777886 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:03.778421 (+ 535us) server_negotiation.cc:299] Negotiation successful
0924 16:06:03.778576 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 16:06:03.779603 13191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:03.766074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40899 (local address 127.25.109.1:47515)
0924 16:06:03.766741 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:03.766783 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:03.767538 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:03.768006 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:03.768013 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:03.768359 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:03.768943 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:03.768968 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:03.769941 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:03.769945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:03.771105 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:03.771117 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:03.771678 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:03.771683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:03.771923 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:03.772673 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:03.772700 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:03.774369 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:03.776516 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:03.776527 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:03.776542 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:03.776824 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:03.777155 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:03.777157 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:03.777159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:03.777361 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:03.777893 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:03.777899 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:03.778159 (+ 260us) client_negotiation.cc:769] Sending connection context
0924 16:06:03.778326 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 16:06:03.778595 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.25.109.2@KRBTEST.COM; defaulting to no policy
I20260924 16:06:03.781045 13189 heartbeater.cc:347] Connected to a master server at 127.25.109.62:40899
I20260924 16:06:03.781340 13189 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:03.781878 13189 heartbeater.cc:511] Master 127.25.109.62:40899 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.2@KRBTEST.COM" created.
I20260924 16:06:03.783605 12997 ts_manager.cc:194] Registered new tserver with Master: 9506d35a969143d7bb420782966b9a38 (127.25.109.1:43483)
I20260924 16:06:03.784865 12997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.1@KRBTEST.COM'} at 127.25.109.1:47515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
Entry for principal kudu/127.25.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab.
I20260924 16:06:03.799991 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:0
--local_ip_for_outbound_sockets=127.25.109.2
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:40899
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:40641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:03.914608 13194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:03.914947 13194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:03.915036 13194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:03.919343 13194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:03.919466 13194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:03.919569 13194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:06:03.925565 13194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:40641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.2.keytab
--principal=kudu/127.25.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:40899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:03.926779 13194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:03.927933 13194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:03.938359 13202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:03.938374 13199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:03.938370 13200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:03.938565 13194 server_base.cc:1034] running on GCE node
I20260924 16:06:03.939131 13194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:03.939800 13194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:03.940963 13194 hybrid_clock.cc:648] HybridClock initialized: now 1790265963940944 us; error 28 us; skew 500 ppm
Sep 24 16:06:03 dist-test-slave-n106 krb5kdc[12951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265963, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:03.943873 13194 init.cc:377] Logged in from keytab as kudu/127.25.109.2@KRBTEST.COM (short username kudu)
I20260924 16:06:03.944864 13194 webserver.cc:460] Webserver started at http://127.25.109.2:38225/ using document root <none> and password file <none>
I20260924 16:06:03.945464 13194 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:03.945528 13194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:03.945791 13194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:03.947919 13194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-1/data/instance:
uuid: "1194b0276db146f19ab45016fd755927"
format_stamp: "Formatted at 2026-09-24 16:06:03 on dist-test-slave-n106"
server_key: "9c788922fceb0d8b79b5ed1891438fca"
server_key_iv: "a2e2a4d5c7ad8c029a2d92c7089d3d8b"
server_key_version: "encryptionkey@0"
I20260924 16:06:03.948592 13194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance:
uuid: "1194b0276db146f19ab45016fd755927"
format_stamp: "Formatted at 2026-09-24 16:06:03 on dist-test-slave-n106"
server_key: "9c788922fceb0d8b79b5ed1891438fca"
server_key_iv: "a2e2a4d5c7ad8c029a2d92c7089d3d8b"
server_key_version: "encryptionkey@0"
I20260924 16:06:03.952711 13194 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:06:03.955309 13209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:03.956701 13194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:06:03.956802 13194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "1194b0276db146f19ab45016fd755927"
format_stamp: "Formatted at 2026-09-24 16:06:03 on dist-test-slave-n106"
server_key: "9c788922fceb0d8b79b5ed1891438fca"
server_key_iv: "a2e2a4d5c7ad8c029a2d92c7089d3d8b"
server_key_version: "encryptionkey@0"
I20260924 16:06:03.956894 13194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:03.971973 13194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:03.975263 13194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:03.975441 13194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:03.976161 13194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:06:03.977207 13194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:03.977259 13194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:03.977303 13194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:03.977324 13194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:03.986143 13194 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:34289
I20260924 16:06:03.986162 13325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:34289 every 8 connection(s)
I20260924 16:06:03.987179 13194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
Sep 24 16:06:03 dist-test-slave-n106 krb5kdc[12951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265963, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.2@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:06:03.996716 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 13194
I20260924 16:06:03.996845 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance
I20260924 16:06:03.997154 26036 external_mini_cluster.cc:1428] Setting key b652a308d6c127a1539fc732bb69a5e0
I20260924 16:06:04.001407 13036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:03.989198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:51011 (local address 127.25.109.62:40899)
0924 16:06:03.989379 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:03.989385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:03.990483 (+ 1098us) server_negotiation.cc:407] Connection header received
0924 16:06:03.991320 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:03.991322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:03.991370 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:03.991446 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:03.992658 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:03.993209 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:03.994394 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:03.994616 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:03.994700 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:03.997383 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:03.997386 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:03.997388 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:03.997413 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:03.999084 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:03.999586 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:03.999592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:03.999597 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:03.999696 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:04.000024 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:04.000030 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:04.000034 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:04.000307 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:04.000440 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:04.001166 (+ 726us) server_negotiation.cc:299] Negotiation successful
0924 16:06:04.001278 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 16:06:04.002751 13328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:03.989642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40899 (local address 127.25.109.2:51011)
0924 16:06:03.990321 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:03.990364 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:03.991137 (+ 773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:03.991630 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:03.991637 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:03.991994 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:03.992523 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:03.992537 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:03.993319 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:03.993321 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:03.994279 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:03.994289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:03.994754 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:03.994757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:03.994903 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:03.995505 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:03.995521 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:03.997170 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:03.999208 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:03.999213 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:03.999227 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:03.999484 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:03.999789 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:03.999791 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:03.999793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:03.999931 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:04.000446 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:04.000455 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:04.000876 (+ 421us) client_negotiation.cc:769] Sending connection context
0924 16:06:04.001117 (+ 241us) client_negotiation.cc:240] Negotiation successful
0924 16:06:04.001519 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":124,"threads_started":1}
I20260924 16:06:04.004032 13326 heartbeater.cc:347] Connected to a master server at 127.25.109.62:40899
I20260924 16:06:04.004323 13326 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:04.004889 13326 heartbeater.cc:511] Master 127.25.109.62:40899 requested a full tablet report, sending...
I20260924 16:06:04.006057 12997 ts_manager.cc:194] Registered new tserver with Master: 1194b0276db146f19ab45016fd755927 (127.25.109.2:34289)
I20260924 16:06:04.006667 12997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.2@KRBTEST.COM'} at 127.25.109.2:51011
WARNING: no policy specified for kudu/127.25.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
Entry for principal kudu/127.25.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab.
I20260924 16:06:04.026957 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:0
--local_ip_for_outbound_sockets=127.25.109.3
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:40899
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:40641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:04.144331 13331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:04.144706 13331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:04.144788 13331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:04.148885 13331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:04.148963 13331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:04.149051 13331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:06:04.154633 13331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:40641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.3.keytab
--principal=kudu/127.25.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:40899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:04.155715 13331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:04.156836 13331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:04.167219 13336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:04.167272 13337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:04.167323 13339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:04.167268 13331 server_base.cc:1034] running on GCE node
I20260924 16:06:04.167943 13331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:04.168678 13331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:04.169850 13331 hybrid_clock.cc:648] HybridClock initialized: now 1790265964169828 us; error 31 us; skew 500 ppm
Sep 24 16:06:04 dist-test-slave-n106 krb5kdc[12951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265964, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:04.172701 13331 init.cc:377] Logged in from keytab as kudu/127.25.109.3@KRBTEST.COM (short username kudu)
I20260924 16:06:04.173554 13331 webserver.cc:460] Webserver started at http://127.25.109.3:41339/ using document root <none> and password file <none>
I20260924 16:06:04.174145 13331 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:04.174235 13331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:04.174472 13331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:04.176645 13331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-2/data/instance:
uuid: "0c914f36e99545b4830a46239e90f138"
format_stamp: "Formatted at 2026-09-24 16:06:04 on dist-test-slave-n106"
server_key: "ecd33c1e036f78a5507594858def24f5"
server_key_iv: "1dff60b6c260f36954ccf85424cc78ec"
server_key_version: "encryptionkey@0"
I20260924 16:06:04.177241 13331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance:
uuid: "0c914f36e99545b4830a46239e90f138"
format_stamp: "Formatted at 2026-09-24 16:06:04 on dist-test-slave-n106"
server_key: "ecd33c1e036f78a5507594858def24f5"
server_key_iv: "1dff60b6c260f36954ccf85424cc78ec"
server_key_version: "encryptionkey@0"
I20260924 16:06:04.181120 13331 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:06:04.183569 13346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:04.184868 13331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:06:04.185001 13331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "0c914f36e99545b4830a46239e90f138"
format_stamp: "Formatted at 2026-09-24 16:06:04 on dist-test-slave-n106"
server_key: "ecd33c1e036f78a5507594858def24f5"
server_key_iv: "1dff60b6c260f36954ccf85424cc78ec"
server_key_version: "encryptionkey@0"
I20260924 16:06:04.185117 13331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:04.241444 13331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:04.244791 13331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:04.245014 13331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:04.245695 13331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:06:04.246605 13331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:04.246680 13331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:04.246755 13331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:04.246806 13331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:04.256028 13331 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:33427
I20260924 16:06:04.256049 13462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:33427 every 8 connection(s)
I20260924 16:06:04.256987 13331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:06:04.264402 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 13331
I20260924 16:06:04.264575 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance
I20260924 16:06:04.264945 26036 external_mini_cluster.cc:1428] Setting key c6f916342945528f7a5fbeafa7c50edf
Sep 24 16:06:04 dist-test-slave-n106 krb5kdc[12951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265964, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.3@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:06:04.271036 13036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:04.258986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:58737 (local address 127.25.109.62:40899)
0924 16:06:04.259180 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:04.259185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:04.260299 (+ 1114us) server_negotiation.cc:407] Connection header received
0924 16:06:04.261303 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:04.261306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:04.261356 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:04.261431 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:04.262619 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.263170 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.264338 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.264732 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.264845 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:04.267844 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:04.267847 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:04.267849 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:04.267876 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:04.269226 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:04.269687 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:04.269690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:04.269691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:04.269765 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:04.270055 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:04.270057 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:04.270058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:04.270233 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:04.270308 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:04.270813 (+ 505us) server_negotiation.cc:299] Negotiation successful
0924 16:06:04.270914 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 16:06:04.271926 13465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:04.259357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:40899 (local address 127.25.109.3:58737)
0924 16:06:04.260143 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:04.260189 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:04.261069 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:04.261628 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:04.261635 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:04.261990 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:04.262489 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:04.262504 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.263278 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.263280 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:04.264206 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:04.264216 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.264921 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.264925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:04.265142 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:04.265824 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:04.265851 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:04.267695 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:04.269327 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:04.269332 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:04.269344 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:04.269599 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:04.269845 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:04.269847 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:04.269848 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:04.269977 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:04.270319 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:04.270324 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:04.270608 (+ 284us) client_negotiation.cc:769] Sending connection context
0924 16:06:04.270767 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 16:06:04.271019 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":122,"threads_started":1}
I20260924 16:06:04.273066 13463 heartbeater.cc:347] Connected to a master server at 127.25.109.62:40899
I20260924 16:06:04.273319 13463 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:04.273833 13463 heartbeater.cc:511] Master 127.25.109.62:40899 requested a full tablet report, sending...
I20260924 16:06:04.274863 12997 ts_manager.cc:194] Registered new tserver with Master: 0c914f36e99545b4830a46239e90f138 (127.25.109.3:33427)
I20260924 16:06:04.275439 12997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.109.3@KRBTEST.COM'} at 127.25.109.3:58737
I20260924 16:06:04.279045 26036 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:06:04.288345 13036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:04.281168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53598 (local address 127.25.109.62:40899)
0924 16:06:04.281297 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:04.281307 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:04.281508 (+ 201us) server_negotiation.cc:407] Connection header received
0924 16:06:04.281616 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:04.281618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:04.281656 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:04.281722 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:04.282384 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.282876 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.283686 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.283897 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.283969 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:04.285355 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:04.285357 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:04.285359 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:04.285379 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:04.286574 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:04.287054 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:04.287056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:04.287057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:04.287148 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:04.287447 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:04.287449 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:04.287450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:04.287676 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:04.287785 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:04.288123 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 16:06:04.288224 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 16:06:04.291473 12997 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:06:04.294772 12997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:06:04.309373 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:04.303570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:43483 (local address 127.0.0.1:39216)
0924 16:06:04.303978 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:04.304036 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:04.304186 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:04.305018 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:04.305024 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:04.305049 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:04.305228 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:04.305238 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.306637 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.306644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:04.307732 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:04.307744 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.308796 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.308800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:04.308914 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:04.308951 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 16:06:04.309065 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 16:06:04.309190 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":45,"thread_start_us":116,"threads_started":1}
I20260924 16:06:04.309521 13476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:04.303838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:34289 (local address 127.0.0.1:33510)
0924 16:06:04.304613 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:04.304628 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:04.304711 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:04.305403 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:04.305405 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:04.305424 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:04.305531 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:04.305538 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.306718 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.306722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:04.307942 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:04.307952 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.309093 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.309096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:04.309186 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:04.309206 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 16:06:04.309294 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 16:06:04.309348 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":101,"threads_started":1}
I20260924 16:06:04.309588 13475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:04.303731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39216 (local address 127.25.109.1:43483)
0924 16:06:04.304166 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:04.304173 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:04.304197 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:06:04.304400 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:04.304429 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:04.304677 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:04.304830 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:04.305361 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.306510 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.307869 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.308667 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.308788 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:04.308948 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:04.309334 (+ 386us) server_negotiation.cc:299] Negotiation successful
0924 16:06:04.309440 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":123,"threads_started":1}
I20260924 16:06:04.310063 13478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:04.304486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33510 (local address 127.25.109.2:34289)
0924 16:06:04.304951 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:04.304963 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:04.304988 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:06:04.305072 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:04.305078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:04.305272 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:04.305431 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:04.305636 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.306554 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.308094 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.308970 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.309073 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:04.309226 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:04.309870 (+ 644us) server_negotiation.cc:299] Negotiation successful
0924 16:06:04.309956 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":159,"threads_started":1}
I20260924 16:06:04.310595 13477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:04.304377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:33427 (local address 127.0.0.1:44362)
0924 16:06:04.305095 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:04.305108 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:04.305188 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:04.306062 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:04.306065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:04.306085 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:04.306202 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:04.306209 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.307659 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.307662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:04.308801 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:04.308813 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.310194 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.310198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:04.310298 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:04.310324 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 16:06:04.310373 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 16:06:04.310446 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":119,"threads_started":1}
I20260924 16:06:04.311242 13479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:04.304943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44362 (local address 127.25.109.3:33427)
0924 16:06:04.305623 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:04.305634 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:04.305656 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:06:04.305733 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:04.305737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:04.305905 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:04.306052 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:04.306324 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.307494 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.308954 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.310054 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.310904 (+ 850us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:04.311017 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:04.311058 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 16:06:04.311131 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":96,"threads_started":1}
I20260924 16:06:04.312950 13123 tablet_service.cc:1467] Processing CreateTablet for tablet a55797c2fe274c28a67a4d5c8828517c (DEFAULT_TABLE table=test-table [id=7b789f9b79ca4edf8dc9732644d74d83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:04.312949 13258 tablet_service.cc:1467] Processing CreateTablet for tablet a55797c2fe274c28a67a4d5c8828517c (DEFAULT_TABLE table=test-table [id=7b789f9b79ca4edf8dc9732644d74d83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:04.314178 13123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a55797c2fe274c28a67a4d5c8828517c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:04.314178 13258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a55797c2fe274c28a67a4d5c8828517c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:04.314245 13395 tablet_service.cc:1467] Processing CreateTablet for tablet a55797c2fe274c28a67a4d5c8828517c (DEFAULT_TABLE table=test-table [id=7b789f9b79ca4edf8dc9732644d74d83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:04.315224 13395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a55797c2fe274c28a67a4d5c8828517c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:04.321404 13480 tablet_bootstrap.cc:492] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138: Bootstrap starting.
I20260924 16:06:04.322312 13482 tablet_bootstrap.cc:492] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927: Bootstrap starting.
I20260924 16:06:04.324362 13482 tablet_bootstrap.cc:654] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:04.324362 13480 tablet_bootstrap.cc:654] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:04.324743 13481 tablet_bootstrap.cc:492] T a55797c2fe274c28a67a4d5c8828517c P 9506d35a969143d7bb420782966b9a38: Bootstrap starting.
I20260924 16:06:04.325829 13482 log.cc:824] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:04.325829 13480 log.cc:824] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:04.326856 13481 tablet_bootstrap.cc:654] T a55797c2fe274c28a67a4d5c8828517c P 9506d35a969143d7bb420782966b9a38: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:04.327715 13481 log.cc:824] T a55797c2fe274c28a67a4d5c8828517c P 9506d35a969143d7bb420782966b9a38: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:04.327840 13482 tablet_bootstrap.cc:492] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927: No bootstrap required, opened a new log
I20260924 16:06:04.327939 13480 tablet_bootstrap.cc:492] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138: No bootstrap required, opened a new log
I20260924 16:06:04.328106 13482 ts_tablet_manager.cc:1397] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 16:06:04.328138 13480 ts_tablet_manager.cc:1397] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 16:06:04.329684 13481 tablet_bootstrap.cc:492] T a55797c2fe274c28a67a4d5c8828517c P 9506d35a969143d7bb420782966b9a38: No bootstrap required, opened a new log
I20260924 16:06:04.329931 13481 ts_tablet_manager.cc:1397] T a55797c2fe274c28a67a4d5c8828517c P 9506d35a969143d7bb420782966b9a38: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 16:06:04.331118 13482 raft_consensus.cc:348] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9506d35a969143d7bb420782966b9a38" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 43483 } } peers { permanent_uuid: "0c914f36e99545b4830a46239e90f138" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 33427 } } peers { permanent_uuid: "1194b0276db146f19ab45016fd755927" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 34289 } }
I20260924 16:06:04.331398 13482 raft_consensus.cc:374] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:04.331473 13482 raft_consensus.cc:729] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1194b0276db146f19ab45016fd755927, State: Initialized, Role: FOLLOWER
I20260924 16:06:04.331969 13482 consensus_queue.cc:260] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9506d35a969143d7bb420782966b9a38" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 43483 } } peers { permanent_uuid: "0c914f36e99545b4830a46239e90f138" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 33427 } } peers { permanent_uuid: "1194b0276db146f19ab45016fd755927" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 34289 } }
I20260924 16:06:04.332170 13480 raft_consensus.cc:348] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9506d35a969143d7bb420782966b9a38" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 43483 } } peers { permanent_uuid: "0c914f36e99545b4830a46239e90f138" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 33427 } } peers { permanent_uuid: "1194b0276db146f19ab45016fd755927" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 34289 } }
I20260924 16:06:04.332497 13480 raft_consensus.cc:374] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:04.332784 13480 raft_consensus.cc:729] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c914f36e99545b4830a46239e90f138, State: Initialized, Role: FOLLOWER
I20260924 16:06:04.332804 13326 heartbeater.cc:503] Master 127.25.109.62:40899 was elected leader, sending a full tablet report...
I20260924 16:06:04.333369 13482 ts_tablet_manager.cc:1428] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 16:06:04.333647 13480 consensus_queue.cc:260] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9506d35a969143d7bb420782966b9a38" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 43483 } } peers { permanent_uuid: "0c914f36e99545b4830a46239e90f138" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 33427 } } peers { permanent_uuid: "1194b0276db146f19ab45016fd755927" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 34289 } }
I20260924 16:06:04.334641 13480 ts_tablet_manager.cc:1428] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 16:06:04.334959 13463 heartbeater.cc:503] Master 127.25.109.62:40899 was elected leader, sending a full tablet report...
I20260924 16:06:04.335707 13481 raft_consensus.cc:348] T a55797c2fe274c28a67a4d5c8828517c P 9506d35a969143d7bb420782966b9a38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9506d35a969143d7bb420782966b9a38" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 43483 } } peers { permanent_uuid: "0c914f36e99545b4830a46239e90f138" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 33427 } } peers { permanent_uuid: "1194b0276db146f19ab45016fd755927" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 34289 } }
I20260924 16:06:04.335958 13481 raft_consensus.cc:374] T a55797c2fe274c28a67a4d5c8828517c P 9506d35a969143d7bb420782966b9a38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:04.336028 13481 raft_consensus.cc:729] T a55797c2fe274c28a67a4d5c8828517c P 9506d35a969143d7bb420782966b9a38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9506d35a969143d7bb420782966b9a38, State: Initialized, Role: FOLLOWER
I20260924 16:06:04.336525 13481 consensus_queue.cc:260] T a55797c2fe274c28a67a4d5c8828517c P 9506d35a969143d7bb420782966b9a38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9506d35a969143d7bb420782966b9a38" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 43483 } } peers { permanent_uuid: "0c914f36e99545b4830a46239e90f138" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 33427 } } peers { permanent_uuid: "1194b0276db146f19ab45016fd755927" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 34289 } }
I20260924 16:06:04.337198 13189 heartbeater.cc:503] Master 127.25.109.62:40899 was elected leader, sending a full tablet report...
I20260924 16:06:04.337519 13481 ts_tablet_manager.cc:1428] T a55797c2fe274c28a67a4d5c8828517c P 9506d35a969143d7bb420782966b9a38: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260924 16:06:04.488483 13327 tablet.cc:2369] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:06:04.508054 13464 tablet.cc:2369] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:06:04.515137 13190 tablet.cc:2369] T a55797c2fe274c28a67a4d5c8828517c P 9506d35a969143d7bb420782966b9a38: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:06:04.592144 13487 raft_consensus.cc:482] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:06:04.592159 13486 raft_consensus.cc:482] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:06:04.592438 13486 raft_consensus.cc:504] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9506d35a969143d7bb420782966b9a38" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 43483 } } peers { permanent_uuid: "0c914f36e99545b4830a46239e90f138" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 33427 } } peers { permanent_uuid: "1194b0276db146f19ab45016fd755927" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 34289 } }
I20260924 16:06:04.592398 13487 raft_consensus.cc:504] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9506d35a969143d7bb420782966b9a38" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 43483 } } peers { permanent_uuid: "0c914f36e99545b4830a46239e90f138" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 33427 } } peers { permanent_uuid: "1194b0276db146f19ab45016fd755927" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 34289 } }
I20260924 16:06:04.593672 13486 leader_election.cc:288] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9506d35a969143d7bb420782966b9a38 (127.25.109.1:43483), 0c914f36e99545b4830a46239e90f138 (127.25.109.3:33427)
I20260924 16:06:04.593675 13487 leader_election.cc:288] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9506d35a969143d7bb420782966b9a38 (127.25.109.1:43483), 1194b0276db146f19ab45016fd755927 (127.25.109.2:34289)
I20260924 16:06:04.599422 13479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:04.594810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:52041 (local address 127.25.109.3:33427)
0924 16:06:04.594905 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:04.594911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:04.595152 (+ 241us) server_negotiation.cc:407] Connection header received
0924 16:06:04.595317 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:04.595320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:04.595404 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:04.595497 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:04.595952 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.596671 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.598178 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.598763 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.598842 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:04.598890 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:04.599204 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 16:06:04.599275 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 16:06:04.599431 13478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:04.594937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:51957 (local address 127.25.109.2:34289)
0924 16:06:04.595043 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:04.595048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:04.595061 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:06:04.595124 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:04.595129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:04.595177 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:04.595256 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:04.595693 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.596540 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.598145 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.598712 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.598801 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:04.598866 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:04.599240 (+ 374us) server_negotiation.cc:299] Negotiation successful
0924 16:06:04.599310 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 16:06:04.599448 13475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:04.594286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:43305 (local address 127.25.109.1:43483)
0924 16:06:04.594656 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:04.594661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:04.594947 (+ 286us) server_negotiation.cc:407] Connection header received
0924 16:06:04.595160 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:04.595165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:04.595252 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:04.595302 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:04.595722 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.596542 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.598194 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.598796 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.598872 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:04.598934 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:04.599252 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 16:06:04.599328 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":286,"spinlock_wait_cycles":2944}
I20260924 16:06:04.599834 13465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:04.593937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:43483 (local address 127.25.109.3:47863)
0924 16:06:04.594294 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:04.594312 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:04.594408 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:04.595549 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:04.595552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:04.595566 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:04.595683 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:04.595690 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.597470 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.597473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:04.598595 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:04.598604 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.599479 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.599482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:04.599586 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:04.599612 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 16:06:04.599662 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 16:06:04.599727 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260}
I20260924 16:06:04.599865 13489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:04.594055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:43483 (local address 127.25.109.2:43305)
0924 16:06:04.594870 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:04.594885 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:04.595028 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:04.595447 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:04.595452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:04.595470 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:04.595586 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:04.595596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.596674 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.596677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:04.598071 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:04.598082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.599018 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.599021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:04.599118 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:04.599140 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 16:06:04.599630 (+ 490us) client_negotiation.cc:240] Negotiation successful
0924 16:06:04.599725 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":76,"thread_start_us":73,"threads_started":1}
I20260924 16:06:04.600176 13490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:04.594008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:34289 (local address 127.25.109.3:51957)
0924 16:06:04.594525 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:04.594538 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:04.594631 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:04.595399 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:04.595403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:04.595424 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:04.595565 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:04.595572 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.596674 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.596677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:04.597990 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:04.598002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.598963 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.598970 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:04.599093 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:04.599118 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 16:06:04.599782 (+ 664us) client_negotiation.cc:240] Negotiation successful
0924 16:06:04.599852 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":3,"thread_start_us":131,"threads_started":1}
I20260924 16:06:04.600178 13491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:04.594385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:33427 (local address 127.25.109.2:52041)
0924 16:06:04.595060 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:04.595075 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:04.595209 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:04.595652 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:04.595657 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:04.595676 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:04.595799 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:04.595805 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.596798 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.596802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:04.598046 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:04.598056 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.598982 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.598985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:04.599071 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:04.599093 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 16:06:04.599610 (+ 517us) client_negotiation.cc:240] Negotiation successful
0924 16:06:04.599899 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":3,"thread_start_us":67,"threads_started":1}
I20260924 16:06:04.600481 13492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:04.594632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:47863 (local address 127.25.109.1:43483)
0924 16:06:04.595094 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:04.595105 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:04.595116 (+ 11us) server_negotiation.cc:407] Connection header received
0924 16:06:04.595158 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:04.595160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:04.595242 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:04.595780 (+ 538us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:04.596549 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.597343 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.598726 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.599352 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.600124 (+ 772us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:04.600174 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:04.600215 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 16:06:04.600290 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":394,"spinlock_wait_cycles":27904,"thread_start_us":119,"threads_started":1}
I20260924 16:06:04.600642 13143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a55797c2fe274c28a67a4d5c8828517c" candidate_uuid: "1194b0276db146f19ab45016fd755927" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9506d35a969143d7bb420782966b9a38" is_pre_election: true
I20260924 16:06:04.600796 13415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a55797c2fe274c28a67a4d5c8828517c" candidate_uuid: "1194b0276db146f19ab45016fd755927" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c914f36e99545b4830a46239e90f138" is_pre_election: true
I20260924 16:06:04.601028 13415 raft_consensus.cc:2454] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1194b0276db146f19ab45016fd755927 in term 0.
I20260924 16:06:04.601029 13143 raft_consensus.cc:2454] T a55797c2fe274c28a67a4d5c8828517c P 9506d35a969143d7bb420782966b9a38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1194b0276db146f19ab45016fd755927 in term 0.
I20260924 16:06:04.601121 13278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a55797c2fe274c28a67a4d5c8828517c" candidate_uuid: "0c914f36e99545b4830a46239e90f138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1194b0276db146f19ab45016fd755927" is_pre_election: true
I20260924 16:06:04.601301 13278 raft_consensus.cc:2454] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c914f36e99545b4830a46239e90f138 in term 0.
I20260924 16:06:04.601369 13142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a55797c2fe274c28a67a4d5c8828517c" candidate_uuid: "0c914f36e99545b4830a46239e90f138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9506d35a969143d7bb420782966b9a38" is_pre_election: true
I20260924 16:06:04.601480 13142 raft_consensus.cc:2454] T a55797c2fe274c28a67a4d5c8828517c P 9506d35a969143d7bb420782966b9a38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c914f36e99545b4830a46239e90f138 in term 0.
I20260924 16:06:04.601527 13212 leader_election.cc:302] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c914f36e99545b4830a46239e90f138, 1194b0276db146f19ab45016fd755927; no voters:
I20260924 16:06:04.601756 13350 leader_election.cc:302] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c914f36e99545b4830a46239e90f138, 1194b0276db146f19ab45016fd755927; no voters:
I20260924 16:06:04.601943 13486 raft_consensus.cc:2781] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:06:04.602002 13487 raft_consensus.cc:2781] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:06:04.602080 13487 raft_consensus.cc:482] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:06:04.602057 13486 raft_consensus.cc:482] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:06:04.602135 13486 raft_consensus.cc:3037] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:04.602130 13487 raft_consensus.cc:3037] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:04.603196 13486 raft_consensus.cc:504] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9506d35a969143d7bb420782966b9a38" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 43483 } } peers { permanent_uuid: "0c914f36e99545b4830a46239e90f138" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 33427 } } peers { permanent_uuid: "1194b0276db146f19ab45016fd755927" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 34289 } }
I20260924 16:06:04.603207 13487 raft_consensus.cc:504] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9506d35a969143d7bb420782966b9a38" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 43483 } } peers { permanent_uuid: "0c914f36e99545b4830a46239e90f138" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 33427 } } peers { permanent_uuid: "1194b0276db146f19ab45016fd755927" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 34289 } }
I20260924 16:06:04.603593 13487 leader_election.cc:288] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138 [CANDIDATE]: Term 1 election: Requested vote from peers 9506d35a969143d7bb420782966b9a38 (127.25.109.1:43483), 1194b0276db146f19ab45016fd755927 (127.25.109.2:34289)
I20260924 16:06:04.603615 13486 leader_election.cc:288] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927 [CANDIDATE]: Term 1 election: Requested vote from peers 9506d35a969143d7bb420782966b9a38 (127.25.109.1:43483), 0c914f36e99545b4830a46239e90f138 (127.25.109.3:33427)
I20260924 16:06:04.603941 13278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a55797c2fe274c28a67a4d5c8828517c" candidate_uuid: "0c914f36e99545b4830a46239e90f138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1194b0276db146f19ab45016fd755927"
I20260924 16:06:04.603952 13142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a55797c2fe274c28a67a4d5c8828517c" candidate_uuid: "0c914f36e99545b4830a46239e90f138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9506d35a969143d7bb420782966b9a38"
I20260924 16:06:04.604020 13143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a55797c2fe274c28a67a4d5c8828517c" candidate_uuid: "1194b0276db146f19ab45016fd755927" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9506d35a969143d7bb420782966b9a38"
I20260924 16:06:04.604066 13142 raft_consensus.cc:3037] T a55797c2fe274c28a67a4d5c8828517c P 9506d35a969143d7bb420782966b9a38 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:04.604118 13278 raft_consensus.cc:2379] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0c914f36e99545b4830a46239e90f138 in current term 1: Already voted for candidate 1194b0276db146f19ab45016fd755927 in this term.
I20260924 16:06:04.604120 13415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a55797c2fe274c28a67a4d5c8828517c" candidate_uuid: "1194b0276db146f19ab45016fd755927" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c914f36e99545b4830a46239e90f138"
I20260924 16:06:04.604282 13415 raft_consensus.cc:2379] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1194b0276db146f19ab45016fd755927 in current term 1: Already voted for candidate 0c914f36e99545b4830a46239e90f138 in this term.
I20260924 16:06:04.604822 13212 leader_election.cc:302] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1194b0276db146f19ab45016fd755927; no voters: 0c914f36e99545b4830a46239e90f138, 9506d35a969143d7bb420782966b9a38
I20260924 16:06:04.605068 13486 raft_consensus.cc:2726] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 16:06:04.605466 13142 raft_consensus.cc:2454] T a55797c2fe274c28a67a4d5c8828517c P 9506d35a969143d7bb420782966b9a38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c914f36e99545b4830a46239e90f138 in term 1.
I20260924 16:06:04.605798 13349 leader_election.cc:302] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c914f36e99545b4830a46239e90f138, 9506d35a969143d7bb420782966b9a38; no voters: 1194b0276db146f19ab45016fd755927
I20260924 16:06:04.605965 13487 raft_consensus.cc:2781] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:04.606180 13487 raft_consensus.cc:686] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138 [term 1 LEADER]: Becoming Leader. State: Replica: 0c914f36e99545b4830a46239e90f138, State: Running, Role: LEADER
I20260924 16:06:04.606508 13487 consensus_queue.cc:237] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9506d35a969143d7bb420782966b9a38" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 43483 } } peers { permanent_uuid: "0c914f36e99545b4830a46239e90f138" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 33427 } } peers { permanent_uuid: "1194b0276db146f19ab45016fd755927" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 34289 } }
I20260924 16:06:04.609833 12996 catalog_manager.cc:5495] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c914f36e99545b4830a46239e90f138 (127.25.109.3). New cstate: current_term: 1 leader_uuid: "0c914f36e99545b4830a46239e90f138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9506d35a969143d7bb420782966b9a38" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 43483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c914f36e99545b4830a46239e90f138" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 33427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1194b0276db146f19ab45016fd755927" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 34289 } health_report { overall_health: UNKNOWN } } }
I20260924 16:06:04.674490 13479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:04.670754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44366 (local address 127.25.109.3:33427)
0924 16:06:04.670920 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:04.670925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:04.670940 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:06:04.671096 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:04.671099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:04.671171 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:04.671295 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:06:04.671559 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.672068 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.672962 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.673191 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.673302 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:04.673635 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:06:04.674076 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:06:04.674225 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:04.674260 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 16:06:04.674314 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 16:06:04.680848 13278 raft_consensus.cc:1261] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927 [term 1 FOLLOWER]: Refusing update from remote peer 0c914f36e99545b4830a46239e90f138: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:06:04.680848 13142 raft_consensus.cc:1261] T a55797c2fe274c28a67a4d5c8828517c P 9506d35a969143d7bb420782966b9a38 [term 1 FOLLOWER]: Refusing update from remote peer 0c914f36e99545b4830a46239e90f138: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:06:04.681547 13487 consensus_queue.cc:1035] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1194b0276db146f19ab45016fd755927" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 34289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:06:04.681807 13493 consensus_queue.cc:1035] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9506d35a969143d7bb420782966b9a38" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 43483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:06:04.689713 13499 mvcc.cc:205] Tried to move back new op lower bound from 7332929391324700672 to 7332929391030898688. Current Snapshot: MvccSnapshot[applied={T|T < 7332929391324700672}]
I20260924 16:06:04.691269 13498 mvcc.cc:205] Tried to move back new op lower bound from 7332929391324700672 to 7332929391030898688. Current Snapshot: MvccSnapshot[applied={T|T < 7332929391324700672}]
I20260924 16:06:04.698995 12996 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:06:04.699148 12996 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:06:04.701644 12996 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2da08b3aa11b478e92c25a790092ed02: Sending DeleteTablet for 3 replicas of tablet a55797c2fe274c28a67a4d5c8828517c
I20260924 16:06:04.702517 13395 tablet_service.cc:1514] Processing DeleteTablet for tablet a55797c2fe274c28a67a4d5c8828517c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:04 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:44362
I20260924 16:06:04.702584 13258 tablet_service.cc:1514] Processing DeleteTablet for tablet a55797c2fe274c28a67a4d5c8828517c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:04 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:33510
I20260924 16:06:04.702576 13123 tablet_service.cc:1514] Processing DeleteTablet for tablet a55797c2fe274c28a67a4d5c8828517c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:04 UTC) from {username='kudu', principal='kudu/127.25.109.62@KRBTEST.COM'} at 127.0.0.1:39216
I20260924 16:06:04.703290 13508 tablet_replica.cc:330] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927: stopping tablet replica
I20260924 16:06:04.703714 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 13059
I20260924 16:06:04.704811 13508 raft_consensus.cc:2229] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:06:04.705132 13508 raft_consensus.cc:2258] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:04.705941 13507 tablet_replica.cc:330] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138: stopping tablet replica
I20260924 16:06:04.706230 13507 raft_consensus.cc:2229] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:06:04.706594 13507 raft_consensus.cc:2258] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:04.706861 13508 ts_tablet_manager.cc:1905] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:04.707986 13507 ts_tablet_manager.cc:1905] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:04.710466 13508 ts_tablet_manager.cc:1918] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:06:04.710641 13508 log.cc:1196] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-1/wal/wals/a55797c2fe274c28a67a4d5c8828517c
I20260924 16:06:04.710847 13507 ts_tablet_manager.cc:1918] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:06:04.710949 13507 log.cc:1196] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265708917108-26036-0/minicluster-data/ts-2/wal/wals/a55797c2fe274c28a67a4d5c8828517c
I20260924 16:06:04.711192 13508 ts_tablet_manager.cc:1939] T a55797c2fe274c28a67a4d5c8828517c P 1194b0276db146f19ab45016fd755927: Deleting consensus metadata
I20260924 16:06:04.711261 13507 ts_tablet_manager.cc:1939] T a55797c2fe274c28a67a4d5c8828517c P 0c914f36e99545b4830a46239e90f138: Deleting consensus metadata
I20260924 16:06:04.712546 12984 catalog_manager.cc:4841] TS 1194b0276db146f19ab45016fd755927 (127.25.109.2:34289): tablet a55797c2fe274c28a67a4d5c8828517c (table test-table [id=7b789f9b79ca4edf8dc9732644d74d83]) successfully deleted
I20260924 16:06:04.712986 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 13194
W20260924 16:06:04.712986 12983 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.109.1:43483 (error 108)
I20260924 16:06:04.714808 12983 catalog_manager.cc:4841] TS 0c914f36e99545b4830a46239e90f138 (127.25.109.3:33427): tablet a55797c2fe274c28a67a4d5c8828517c (table test-table [id=7b789f9b79ca4edf8dc9732644d74d83]) successfully deleted
I20260924 16:06:04.715849 13477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:04.715505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:43483 (local address 127.0.0.1:39226)
0924 16:06:04.715677 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:04.715757 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.1:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260924 16:06:04.716028 12983 catalog_manager.cc:4584] TS 9506d35a969143d7bb420782966b9a38 (127.25.109.1:43483): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a55797c2fe274c28a67a4d5c8828517c: Network error: Client connection negotiation failed: client connection to 127.25.109.1:43483: connect: Connection refused (error 111)
I20260924 16:06:04.720852 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 13331
I20260924 16:06:04.726677 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 12965
2026-09-24T16:06:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2676 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T16:06:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:04Z Disabled control of system clock
I20260924 16:06:04.757296 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:38473
--webserver_interface=127.25.109.62
--webserver_port=0
--builtin_ntp_servers=127.25.109.20:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:38473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 16:06:04.869704 13514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:04.869987 13514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:04.870023 13514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:04.874136 13514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:04.874198 13514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:04.874213 13514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:04.874230 13514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:04.874246 13514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:04.879040 13514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:38473
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:38473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:04.880151 13514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:04.881587 13514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:04.890365 13523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:04.890354 13519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:04.890461 13514 server_base.cc:1034] running on GCE node
W20260924 16:06:04.890358 13520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:04.891217 13514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:04.892266 13514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:04.893427 13514 hybrid_clock.cc:648] HybridClock initialized: now 1790265964893399 us; error 42 us; skew 500 ppm
I20260924 16:06:04.895325 13514 webserver.cc:460] Webserver started at http://127.25.109.62:38675/ using document root <none> and password file <none>
I20260924 16:06:04.895977 13514 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:04.896046 13514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:04.896283 13514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:04.898448 13514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "d0f2a4c66c9247e49457a7884e87daf9"
format_stamp: "Formatted at 2026-09-24 16:06:04 on dist-test-slave-n106"
server_key: "d56db6013b721482aac48e14200a21ab"
server_key_iv: "f16e2b2d5311cff3fae21684ade35435"
server_key_version: "encryptionkey@0"
I20260924 16:06:04.899041 13514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "d0f2a4c66c9247e49457a7884e87daf9"
format_stamp: "Formatted at 2026-09-24 16:06:04 on dist-test-slave-n106"
server_key: "d56db6013b721482aac48e14200a21ab"
server_key_iv: "f16e2b2d5311cff3fae21684ade35435"
server_key_version: "encryptionkey@0"
I20260924 16:06:04.902827 13514 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260924 16:06:04.905347 13529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:04.906608 13514 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:06:04.906766 13514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "d0f2a4c66c9247e49457a7884e87daf9"
format_stamp: "Formatted at 2026-09-24 16:06:04 on dist-test-slave-n106"
server_key: "d56db6013b721482aac48e14200a21ab"
server_key_iv: "f16e2b2d5311cff3fae21684ade35435"
server_key_version: "encryptionkey@0"
I20260924 16:06:04.906883 13514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:04.923432 13514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:04.924181 13514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:04.924387 13514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:04.931025 13514 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:38473
I20260924 16:06:04.931059 13583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:38473 every 8 connection(s)
I20260924 16:06:04.931977 13514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:04.933207 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 13514
I20260924 16:06:04.933341 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:06:04.933687 26036 external_mini_cluster.cc:1428] Setting key ff479c2b11583ea880eea43e0a200b81
I20260924 16:06:04.935875 13584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:04.941527 13584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0f2a4c66c9247e49457a7884e87daf9: Bootstrap starting.
I20260924 16:06:04.943794 13584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0f2a4c66c9247e49457a7884e87daf9: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:04.944619 13584 log.cc:824] T 00000000000000000000000000000000 P d0f2a4c66c9247e49457a7884e87daf9: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:04.946506 13584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0f2a4c66c9247e49457a7884e87daf9: No bootstrap required, opened a new log
I20260924 16:06:04.950214 13584 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0f2a4c66c9247e49457a7884e87daf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f2a4c66c9247e49457a7884e87daf9" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 38473 } }
I20260924 16:06:04.950433 13584 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0f2a4c66c9247e49457a7884e87daf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:04.950495 13584 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0f2a4c66c9247e49457a7884e87daf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0f2a4c66c9247e49457a7884e87daf9, State: Initialized, Role: FOLLOWER
I20260924 16:06:04.950510 13587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:04.935210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32782 (local address 127.25.109.62:38473)
0924 16:06:04.935863 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:04.935878 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:04.935939 (+ 61us) server_negotiation.cc:407] Connection header received
0924 16:06:04.936711 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:04.936731 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:04.937138 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:04.937481 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:04.937907 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.938869 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.939784 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:04.940259 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:04.940459 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:04.948620 (+ 8161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:04.948625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:04.948640 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:04.949099 (+ 459us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:04.949166 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:04.949188 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:04.949296 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:04.949464 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 16:06:04.949738 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":153,"threads_started":1}
I20260924 16:06:04.950959 13584 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0f2a4c66c9247e49457a7884e87daf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f2a4c66c9247e49457a7884e87daf9" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 38473 } }
I20260924 16:06:04.951133 13584 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0f2a4c66c9247e49457a7884e87daf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:04.951215 13584 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0f2a4c66c9247e49457a7884e87daf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:04.951340 13584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0f2a4c66c9247e49457a7884e87daf9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:04.952489 13584 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0f2a4c66c9247e49457a7884e87daf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f2a4c66c9247e49457a7884e87daf9" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 38473 } }
I20260924 16:06:04.952854 13584 leader_election.cc:302] T 00000000000000000000000000000000 P d0f2a4c66c9247e49457a7884e87daf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0f2a4c66c9247e49457a7884e87daf9; no voters:
I20260924 16:06:04.953213 13584 leader_election.cc:288] T 00000000000000000000000000000000 P d0f2a4c66c9247e49457a7884e87daf9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:06:04.953326 13589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0f2a4c66c9247e49457a7884e87daf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:04.953564 13589 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0f2a4c66c9247e49457a7884e87daf9 [term 1 LEADER]: Becoming Leader. State: Replica: d0f2a4c66c9247e49457a7884e87daf9, State: Running, Role: LEADER
I20260924 16:06:04.953866 13589 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0f2a4c66c9247e49457a7884e87daf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f2a4c66c9247e49457a7884e87daf9" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 38473 } }
I20260924 16:06:04.954447 13584 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0f2a4c66c9247e49457a7884e87daf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:04.955554 13591 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0f2a4c66c9247e49457a7884e87daf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0f2a4c66c9247e49457a7884e87daf9. Latest consensus state: current_term: 1 leader_uuid: "d0f2a4c66c9247e49457a7884e87daf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f2a4c66c9247e49457a7884e87daf9" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 38473 } } }
I20260924 16:06:04.955696 13591 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0f2a4c66c9247e49457a7884e87daf9 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:04.955542 13590 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0f2a4c66c9247e49457a7884e87daf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0f2a4c66c9247e49457a7884e87daf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f2a4c66c9247e49457a7884e87daf9" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 38473 } } }
I20260924 16:06:04.955837 13590 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0f2a4c66c9247e49457a7884e87daf9 [sys.catalog]: This master's current role is: LEADER
W20260924 16:06:04.959829 13605 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0f2a4c66c9247e49457a7884e87daf9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:06:04.959923 13605 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:06:04.960001 13604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:04.960819 13604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:04.966652 13604 catalog_manager.cc:1347] Generated new cluster ID: 0544d609c23746f38563677b85e65987
I20260924 16:06:04.966725 13604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:04.981153 13604 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:04.982067 13604 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:04.987054 13604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0f2a4c66c9247e49457a7884e87daf9: Generated new TSK 0
I20260924 16:06:04.987886 13604 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:06:04.998492 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.1:0
--local_ip_for_outbound_sockets=127.25.109.1
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:38473
--builtin_ntp_servers=127.25.109.20:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 16:06:05.113328 13608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:05.113632 13608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:05.113734 13608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:05.117766 13608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:05.117851 13608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:05.117945 13608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.1
I20260924 16:06:05.122741 13608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.109.1
--webserver_port=0
--tserver_master_addrs=127.25.109.62:38473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.1
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:05.123908 13608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:05.125344 13608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:05.134924 13616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:05.134982 13614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:05.135022 13613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:05.135264 13608 server_base.cc:1034] running on GCE node
I20260924 16:06:05.135696 13608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:05.136334 13608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:05.137523 13608 hybrid_clock.cc:648] HybridClock initialized: now 1790265965137494 us; error 39 us; skew 500 ppm
I20260924 16:06:05.139341 13608 webserver.cc:460] Webserver started at http://127.25.109.1:33105/ using document root <none> and password file <none>
I20260924 16:06:05.139943 13608 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:05.140038 13608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:05.140287 13608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:05.142396 13608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-0/data/instance:
uuid: "98afcc80e1af45298f2af1a204b4dbca"
format_stamp: "Formatted at 2026-09-24 16:06:05 on dist-test-slave-n106"
server_key: "c5cc8b8d0621b5e0782a2dfbec0009e7"
server_key_iv: "c4524f944809d1662d93d13b9a206fa5"
server_key_version: "encryptionkey@0"
I20260924 16:06:05.142989 13608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance:
uuid: "98afcc80e1af45298f2af1a204b4dbca"
format_stamp: "Formatted at 2026-09-24 16:06:05 on dist-test-slave-n106"
server_key: "c5cc8b8d0621b5e0782a2dfbec0009e7"
server_key_iv: "c4524f944809d1662d93d13b9a206fa5"
server_key_version: "encryptionkey@0"
I20260924 16:06:05.146831 13608 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:06:05.149529 13622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:05.150859 13608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:06:05.151010 13608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-0/wal
uuid: "98afcc80e1af45298f2af1a204b4dbca"
format_stamp: "Formatted at 2026-09-24 16:06:05 on dist-test-slave-n106"
server_key: "c5cc8b8d0621b5e0782a2dfbec0009e7"
server_key_iv: "c4524f944809d1662d93d13b9a206fa5"
server_key_version: "encryptionkey@0"
I20260924 16:06:05.151140 13608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:05.172374 13608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:05.173153 13608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:05.173364 13608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:05.174005 13608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:06:05.175012 13608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:05.175086 13608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:05.175163 13608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:05.175206 13608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:05.184252 13608 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.1:42323
I20260924 16:06:05.184270 13737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.1:42323 every 8 connection(s)
I20260924 16:06:05.185266 13608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-0/data/info.pb
I20260924 16:06:05.194530 13587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.187242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:52903 (local address 127.25.109.62:38473)
0924 16:06:05.187418 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:05.187422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:05.188540 (+ 1118us) server_negotiation.cc:407] Connection header received
0924 16:06:05.189975 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:05.189978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:05.190032 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:05.190108 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:05.191022 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.191556 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.192756 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.192971 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.193048 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:05.194090 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:05.194093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:05.194095 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:05.194107 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:05.194134 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:05.194142 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:05.194198 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:05.194339 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 16:06:05.194411 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 16:06:05.195240 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 13608
I20260924 16:06:05.195393 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-0/wal/instance
I20260924 16:06:05.195396 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.187641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:38473 (local address 127.25.109.1:52903)
0924 16:06:05.188324 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:05.188367 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:05.189770 (+ 1403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:05.190295 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:05.190302 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:05.190391 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:05.190898 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:05.190911 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.191666 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.191669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:05.192649 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:05.192660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.193114 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.193117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:05.193257 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:05.193839 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:06:05.193841 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:05.193881 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:06:05.193948 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:05.194214 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:05.194221 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:05.194225 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 16:06:05.194320 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 16:06:05.194543 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":107,"threads_started":1}
I20260924 16:06:05.195797 26036 external_mini_cluster.cc:1428] Setting key efe6a1a72c0b9fca520007d1c62a23cd
I20260924 16:06:05.196841 13738 heartbeater.cc:347] Connected to a master server at 127.25.109.62:38473
I20260924 16:06:05.197095 13738 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:05.197739 13738 heartbeater.cc:511] Master 127.25.109.62:38473 requested a full tablet report, sending...
I20260924 16:06:05.197829 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.2:0
--local_ip_for_outbound_sockets=127.25.109.2
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:38473
--builtin_ntp_servers=127.25.109.20:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 16:06:05.199374 13545 ts_manager.cc:194] Registered new tserver with Master: 98afcc80e1af45298f2af1a204b4dbca (127.25.109.1:42323)
I20260924 16:06:05.200752 13545 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.1:52903
W20260924 16:06:05.310541 13741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:05.310851 13741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:05.310891 13741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:05.314873 13741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:05.314929 13741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:05.314981 13741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.2
I20260924 16:06:05.319594 13741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.109.2
--webserver_port=0
--tserver_master_addrs=127.25.109.62:38473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.2
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:05.320657 13741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:05.321973 13741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:05.331312 13746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:05.331396 13749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:05.331563 13741 server_base.cc:1034] running on GCE node
W20260924 16:06:05.331323 13747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:05.332101 13741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:05.332810 13741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:05.333997 13741 hybrid_clock.cc:648] HybridClock initialized: now 1790265965333967 us; error 43 us; skew 500 ppm
I20260924 16:06:05.335765 13741 webserver.cc:460] Webserver started at http://127.25.109.2:46709/ using document root <none> and password file <none>
I20260924 16:06:05.336344 13741 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:05.336455 13741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:05.336704 13741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:05.338734 13741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-1/data/instance:
uuid: "fbb208e6d56b4d50a674793b25ee0ae9"
format_stamp: "Formatted at 2026-09-24 16:06:05 on dist-test-slave-n106"
server_key: "fce49202c7a3edf67789a7016093f77e"
server_key_iv: "dde43a621e5a54fb62abb8baa22d5c94"
server_key_version: "encryptionkey@0"
I20260924 16:06:05.339305 13741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance:
uuid: "fbb208e6d56b4d50a674793b25ee0ae9"
format_stamp: "Formatted at 2026-09-24 16:06:05 on dist-test-slave-n106"
server_key: "fce49202c7a3edf67789a7016093f77e"
server_key_iv: "dde43a621e5a54fb62abb8baa22d5c94"
server_key_version: "encryptionkey@0"
I20260924 16:06:05.342980 13741 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260924 16:06:05.345465 13755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:05.346675 13741 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:06:05.346812 13741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-1/wal
uuid: "fbb208e6d56b4d50a674793b25ee0ae9"
format_stamp: "Formatted at 2026-09-24 16:06:05 on dist-test-slave-n106"
server_key: "fce49202c7a3edf67789a7016093f77e"
server_key_iv: "dde43a621e5a54fb62abb8baa22d5c94"
server_key_version: "encryptionkey@0"
I20260924 16:06:05.346927 13741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:05.368216 13741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:05.369035 13741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:05.369240 13741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:05.369866 13741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:06:05.370844 13741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:05.370914 13741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:05.370987 13741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:05.371038 13741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:05.380029 13741 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.2:45507
I20260924 16:06:05.380057 13871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.2:45507 every 8 connection(s)
I20260924 16:06:05.380985 13741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-1/data/info.pb
I20260924 16:06:05.384828 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 13741
I20260924 16:06:05.384955 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-1/wal/instance
I20260924 16:06:05.385272 26036 external_mini_cluster.cc:1428] Setting key d6ceb828ed89c7dc5da38d2b4ab9dd54
I20260924 16:06:05.387356 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.3:0
--local_ip_for_outbound_sockets=127.25.109.3
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:38473
--builtin_ntp_servers=127.25.109.20:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 16:06:05.390764 13587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.382832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.2:55563 (local address 127.25.109.62:38473)
0924 16:06:05.382983 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:05.382987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:05.384086 (+ 1099us) server_negotiation.cc:407] Connection header received
0924 16:06:05.385737 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:05.385741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:05.385804 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:05.385883 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:05.386975 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.387559 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.388770 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.389002 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.389139 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:05.390251 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:05.390260 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:05.390262 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:05.390274 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:05.390294 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:05.390302 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:05.390381 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:05.390548 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 16:06:05.390617 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 16:06:05.391768 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.383249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:38473 (local address 127.25.109.2:55563)
0924 16:06:05.383923 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:05.383964 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:05.385507 (+ 1543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:05.386091 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:05.386101 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:05.386212 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:05.386826 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:05.386840 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.387672 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.387676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:05.388643 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:05.388653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.389184 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.389189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:05.389359 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:05.389979 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:06:05.389980 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:05.390024 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:06:05.390092 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:05.390385 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:05.390396 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:05.390404 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 16:06:05.390545 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 16:06:05.390868 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":118,"threads_started":1}
I20260924 16:06:05.393003 13872 heartbeater.cc:347] Connected to a master server at 127.25.109.62:38473
I20260924 16:06:05.393273 13872 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:05.393821 13872 heartbeater.cc:511] Master 127.25.109.62:38473 requested a full tablet report, sending...
I20260924 16:06:05.394971 13545 ts_manager.cc:194] Registered new tserver with Master: fbb208e6d56b4d50a674793b25ee0ae9 (127.25.109.2:45507)
I20260924 16:06:05.395870 13545 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.2:55563
W20260924 16:06:05.501296 13875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:05.501530 13875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:05.501566 13875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:05.505424 13875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:05.505466 13875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:05.505518 13875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.3
I20260924 16:06:05.510042 13875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.109.3
--webserver_port=0
--tserver_master_addrs=127.25.109.62:38473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.3
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:05.511097 13875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:05.512470 13875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:05.522127 13883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:05.522123 13880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:05.522123 13881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:05.522325 13875 server_base.cc:1034] running on GCE node
I20260924 16:06:05.522956 13875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:05.523587 13875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:05.524775 13875 hybrid_clock.cc:648] HybridClock initialized: now 1790265965524762 us; error 33 us; skew 500 ppm
I20260924 16:06:05.526501 13875 webserver.cc:460] Webserver started at http://127.25.109.3:40991/ using document root <none> and password file <none>
I20260924 16:06:05.527125 13875 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:05.527216 13875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:05.527484 13875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:05.529654 13875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-2/data/instance:
uuid: "f49d1db4eb4c4177b29a035b76a1846b"
format_stamp: "Formatted at 2026-09-24 16:06:05 on dist-test-slave-n106"
server_key: "5b003051e207cb641adbbdc7b1ef64df"
server_key_iv: "d274b25c74747cbda90280d352cc7540"
server_key_version: "encryptionkey@0"
I20260924 16:06:05.530146 13875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance:
uuid: "f49d1db4eb4c4177b29a035b76a1846b"
format_stamp: "Formatted at 2026-09-24 16:06:05 on dist-test-slave-n106"
server_key: "5b003051e207cb641adbbdc7b1ef64df"
server_key_iv: "d274b25c74747cbda90280d352cc7540"
server_key_version: "encryptionkey@0"
I20260924 16:06:05.533728 13875 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:06:05.535988 13889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:05.537235 13875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:06:05.537369 13875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-2/wal
uuid: "f49d1db4eb4c4177b29a035b76a1846b"
format_stamp: "Formatted at 2026-09-24 16:06:05 on dist-test-slave-n106"
server_key: "5b003051e207cb641adbbdc7b1ef64df"
server_key_iv: "d274b25c74747cbda90280d352cc7540"
server_key_version: "encryptionkey@0"
I20260924 16:06:05.537485 13875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:05.555938 13875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:05.556775 13875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:05.556978 13875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:05.557571 13875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:06:05.558553 13875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:05.558632 13875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:05.558702 13875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:05.558753 13875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:05.567781 13875 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.3:41789
I20260924 16:06:05.567803 14003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.3:41789 every 8 connection(s)
I20260924 16:06:05.568773 13875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-2/data/info.pb
I20260924 16:06:05.573756 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 13875
I20260924 16:06:05.573890 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-2/wal/instance
I20260924 16:06:05.574203 26036 external_mini_cluster.cc:1428] Setting key 712a1a7bc82de14e30f197ed9bc54ef5
I20260924 16:06:05.578871 13587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.570751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.3:51951 (local address 127.25.109.62:38473)
0924 16:06:05.570932 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:05.570936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:05.572000 (+ 1064us) server_negotiation.cc:407] Connection header received
0924 16:06:05.573569 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:05.573573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:05.573635 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:05.573724 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:05.574918 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.575620 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.576922 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.577154 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.577272 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:05.578362 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:05.578366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:05.578369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:05.578383 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:05.578408 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:05.578423 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:05.578499 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:05.578670 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 16:06:05.578732 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:06:05.579731 14006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.571145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:38473 (local address 127.25.109.3:51951)
0924 16:06:05.571838 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:05.571879 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:05.573305 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:05.573949 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:05.573975 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:05.574088 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:05.574746 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:05.574768 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.575746 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.575750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:05.576789 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:05.576799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.577314 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.577318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:05.577496 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:05.578070 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:06:05.578072 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:05.578115 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:06:05.578181 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:05.578515 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:05.578531 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:05.578536 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 16:06:05.578667 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 16:06:05.578959 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":109,"threads_started":1}
I20260924 16:06:05.580835 14004 heartbeater.cc:347] Connected to a master server at 127.25.109.62:38473
I20260924 16:06:05.581084 14004 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:05.581674 14004 heartbeater.cc:511] Master 127.25.109.62:38473 requested a full tablet report, sending...
I20260924 16:06:05.582875 13545 ts_manager.cc:194] Registered new tserver with Master: f49d1db4eb4c4177b29a035b76a1846b (127.25.109.3:41789)
I20260924 16:06:05.583477 13545 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.3:51951
I20260924 16:06:05.588157 26036 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:06:05.593590 13587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.590166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32788 (local address 127.25.109.62:38473)
0924 16:06:05.590286 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:05.590293 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:05.590443 (+ 150us) server_negotiation.cc:407] Connection header received
0924 16:06:05.590540 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:05.590542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:05.590579 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:05.590648 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:05.590889 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.591378 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.592150 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.592352 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.592495 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:05.593258 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:05.593261 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:05.593263 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:05.593273 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:05.593288 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:05.593295 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:05.593349 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:05.593442 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 16:06:05.593485 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 16:06:05.596148 13545 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:06:05.599531 13545 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:06:05.614080 14016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.608038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:45507 (local address 127.0.0.1:59616)
0924 16:06:05.608497 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:05.608547 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:05.608676 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:05.610064 (+ 1388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:05.610067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:05.610084 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:05.610187 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:05.610193 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.611581 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.611584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:05.612720 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:05.612727 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.613516 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.613520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:05.613636 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:05.613676 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 16:06:05.613793 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 16:06:05.613916 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":23,"spinlock_wait_cycles":10112,"thread_start_us":105,"threads_started":1}
I20260924 16:06:05.614080 14020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.608921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59616 (local address 127.25.109.2:45507)
0924 16:06:05.609553 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:05.609565 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:05.609589 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:06:05.609671 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:05.609676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:05.609883 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:05.610084 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:05.610283 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.611466 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.612837 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.613389 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.613517 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:05.613722 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:05.613822 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 16:06:05.613916 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":112,"threads_started":1}
I20260924 16:06:05.614246 14015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.608058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:42323 (local address 127.0.0.1:54854)
0924 16:06:05.608692 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:05.608705 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:05.608778 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:05.609540 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:05.609546 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:05.609569 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:05.609717 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:05.609767 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.611107 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.611118 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:05.612485 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:05.612496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.613814 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.613818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:05.613911 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:05.613934 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 16:06:05.614021 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 16:06:05.614103 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":94,"threads_started":1}
I20260924 16:06:05.614687 14018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.608954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:41789 (local address 127.0.0.1:44462)
0924 16:06:05.609807 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:05.609822 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:05.609910 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:05.610309 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:05.610312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:05.610327 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:05.610406 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:05.610411 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.611816 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.611822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:05.613026 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:05.613037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.614264 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.614270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:05.614366 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:05.614391 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 16:06:05.614478 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 16:06:05.614541 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"mutex_wait_us":24,"thread_start_us":86,"threads_started":1}
I20260924 16:06:05.614697 14017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.608268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54854 (local address 127.25.109.1:42323)
0924 16:06:05.609029 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:05.609043 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:05.609069 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:06:05.609162 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:05.609167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:05.609393 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:05.609554 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:05.609898 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.610997 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.612617 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.613459 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.613562 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:05.613744 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:05.614441 (+ 697us) server_negotiation.cc:299] Negotiation successful
0924 16:06:05.614546 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":143,"threads_started":1}
I20260924 16:06:05.615109 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.609148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44462 (local address 127.25.109.3:41789)
0924 16:06:05.609813 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:05.609825 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:05.609887 (+ 62us) server_negotiation.cc:407] Connection header received
0924 16:06:05.610015 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:05.610023 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:05.610191 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:05.610340 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:05.610479 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.611683 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.613171 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.613938 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.614036 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:05.614197 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:05.614855 (+ 658us) server_negotiation.cc:299] Negotiation successful
0924 16:06:05.614953 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":158,"threads_started":1}
I20260924 16:06:05.617637 13671 tablet_service.cc:1467] Processing CreateTablet for tablet 15caf455468a46839a03c2cc41554303 (DEFAULT_TABLE table=test-table [id=74db5b07f9a34621ada1ed1b39ea7b70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:05.617769 13804 tablet_service.cc:1467] Processing CreateTablet for tablet 15caf455468a46839a03c2cc41554303 (DEFAULT_TABLE table=test-table [id=74db5b07f9a34621ada1ed1b39ea7b70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:05.617962 13937 tablet_service.cc:1467] Processing CreateTablet for tablet 15caf455468a46839a03c2cc41554303 (DEFAULT_TABLE table=test-table [id=74db5b07f9a34621ada1ed1b39ea7b70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:05.618695 13671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15caf455468a46839a03c2cc41554303. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:05.618809 13804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15caf455468a46839a03c2cc41554303. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:05.618809 13937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15caf455468a46839a03c2cc41554303. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:05.626667 14022 tablet_bootstrap.cc:492] T 15caf455468a46839a03c2cc41554303 P 98afcc80e1af45298f2af1a204b4dbca: Bootstrap starting.
I20260924 16:06:05.626667 14023 tablet_bootstrap.cc:492] T 15caf455468a46839a03c2cc41554303 P fbb208e6d56b4d50a674793b25ee0ae9: Bootstrap starting.
I20260924 16:06:05.628675 14023 tablet_bootstrap.cc:654] T 15caf455468a46839a03c2cc41554303 P fbb208e6d56b4d50a674793b25ee0ae9: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:05.628700 14022 tablet_bootstrap.cc:654] T 15caf455468a46839a03c2cc41554303 P 98afcc80e1af45298f2af1a204b4dbca: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:05.628774 14024 tablet_bootstrap.cc:492] T 15caf455468a46839a03c2cc41554303 P f49d1db4eb4c4177b29a035b76a1846b: Bootstrap starting.
I20260924 16:06:05.629444 14023 log.cc:824] T 15caf455468a46839a03c2cc41554303 P fbb208e6d56b4d50a674793b25ee0ae9: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:05.629494 14022 log.cc:824] T 15caf455468a46839a03c2cc41554303 P 98afcc80e1af45298f2af1a204b4dbca: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:05.630826 14024 tablet_bootstrap.cc:654] T 15caf455468a46839a03c2cc41554303 P f49d1db4eb4c4177b29a035b76a1846b: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:05.631456 14022 tablet_bootstrap.cc:492] T 15caf455468a46839a03c2cc41554303 P 98afcc80e1af45298f2af1a204b4dbca: No bootstrap required, opened a new log
I20260924 16:06:05.631548 14023 tablet_bootstrap.cc:492] T 15caf455468a46839a03c2cc41554303 P fbb208e6d56b4d50a674793b25ee0ae9: No bootstrap required, opened a new log
I20260924 16:06:05.631615 14024 log.cc:824] T 15caf455468a46839a03c2cc41554303 P f49d1db4eb4c4177b29a035b76a1846b: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:05.631724 14022 ts_tablet_manager.cc:1397] T 15caf455468a46839a03c2cc41554303 P 98afcc80e1af45298f2af1a204b4dbca: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 16:06:05.631742 14023 ts_tablet_manager.cc:1397] T 15caf455468a46839a03c2cc41554303 P fbb208e6d56b4d50a674793b25ee0ae9: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 16:06:05.633581 14024 tablet_bootstrap.cc:492] T 15caf455468a46839a03c2cc41554303 P f49d1db4eb4c4177b29a035b76a1846b: No bootstrap required, opened a new log
I20260924 16:06:05.633760 14024 ts_tablet_manager.cc:1397] T 15caf455468a46839a03c2cc41554303 P f49d1db4eb4c4177b29a035b76a1846b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 16:06:05.635284 14022 raft_consensus.cc:348] T 15caf455468a46839a03c2cc41554303 P 98afcc80e1af45298f2af1a204b4dbca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb208e6d56b4d50a674793b25ee0ae9" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 45507 } } peers { permanent_uuid: "98afcc80e1af45298f2af1a204b4dbca" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 42323 } } peers { permanent_uuid: "f49d1db4eb4c4177b29a035b76a1846b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41789 } }
I20260924 16:06:05.635536 14022 raft_consensus.cc:374] T 15caf455468a46839a03c2cc41554303 P 98afcc80e1af45298f2af1a204b4dbca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:05.635624 14022 raft_consensus.cc:729] T 15caf455468a46839a03c2cc41554303 P 98afcc80e1af45298f2af1a204b4dbca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98afcc80e1af45298f2af1a204b4dbca, State: Initialized, Role: FOLLOWER
I20260924 16:06:05.636108 14022 consensus_queue.cc:260] T 15caf455468a46839a03c2cc41554303 P 98afcc80e1af45298f2af1a204b4dbca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb208e6d56b4d50a674793b25ee0ae9" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 45507 } } peers { permanent_uuid: "98afcc80e1af45298f2af1a204b4dbca" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 42323 } } peers { permanent_uuid: "f49d1db4eb4c4177b29a035b76a1846b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41789 } }
I20260924 16:06:05.636026 14023 raft_consensus.cc:348] T 15caf455468a46839a03c2cc41554303 P fbb208e6d56b4d50a674793b25ee0ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb208e6d56b4d50a674793b25ee0ae9" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 45507 } } peers { permanent_uuid: "98afcc80e1af45298f2af1a204b4dbca" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 42323 } } peers { permanent_uuid: "f49d1db4eb4c4177b29a035b76a1846b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41789 } }
I20260924 16:06:05.636267 14023 raft_consensus.cc:374] T 15caf455468a46839a03c2cc41554303 P fbb208e6d56b4d50a674793b25ee0ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:05.636348 14023 raft_consensus.cc:729] T 15caf455468a46839a03c2cc41554303 P fbb208e6d56b4d50a674793b25ee0ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb208e6d56b4d50a674793b25ee0ae9, State: Initialized, Role: FOLLOWER
I20260924 16:06:05.636813 14023 consensus_queue.cc:260] T 15caf455468a46839a03c2cc41554303 P fbb208e6d56b4d50a674793b25ee0ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb208e6d56b4d50a674793b25ee0ae9" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 45507 } } peers { permanent_uuid: "98afcc80e1af45298f2af1a204b4dbca" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 42323 } } peers { permanent_uuid: "f49d1db4eb4c4177b29a035b76a1846b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41789 } }
I20260924 16:06:05.637176 14022 ts_tablet_manager.cc:1428] T 15caf455468a46839a03c2cc41554303 P 98afcc80e1af45298f2af1a204b4dbca: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260924 16:06:05.637509 13738 heartbeater.cc:503] Master 127.25.109.62:38473 was elected leader, sending a full tablet report...
I20260924 16:06:05.637509 13872 heartbeater.cc:503] Master 127.25.109.62:38473 was elected leader, sending a full tablet report...
I20260924 16:06:05.637861 14023 ts_tablet_manager.cc:1428] T 15caf455468a46839a03c2cc41554303 P fbb208e6d56b4d50a674793b25ee0ae9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 16:06:05.637916 14024 raft_consensus.cc:348] T 15caf455468a46839a03c2cc41554303 P f49d1db4eb4c4177b29a035b76a1846b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb208e6d56b4d50a674793b25ee0ae9" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 45507 } } peers { permanent_uuid: "98afcc80e1af45298f2af1a204b4dbca" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 42323 } } peers { permanent_uuid: "f49d1db4eb4c4177b29a035b76a1846b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41789 } }
I20260924 16:06:05.638161 14024 raft_consensus.cc:374] T 15caf455468a46839a03c2cc41554303 P f49d1db4eb4c4177b29a035b76a1846b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:05.638221 14024 raft_consensus.cc:729] T 15caf455468a46839a03c2cc41554303 P f49d1db4eb4c4177b29a035b76a1846b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f49d1db4eb4c4177b29a035b76a1846b, State: Initialized, Role: FOLLOWER
I20260924 16:06:05.638640 14024 consensus_queue.cc:260] T 15caf455468a46839a03c2cc41554303 P f49d1db4eb4c4177b29a035b76a1846b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb208e6d56b4d50a674793b25ee0ae9" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 45507 } } peers { permanent_uuid: "98afcc80e1af45298f2af1a204b4dbca" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 42323 } } peers { permanent_uuid: "f49d1db4eb4c4177b29a035b76a1846b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41789 } }
I20260924 16:06:05.639290 14004 heartbeater.cc:503] Master 127.25.109.62:38473 was elected leader, sending a full tablet report...
I20260924 16:06:05.640275 14024 ts_tablet_manager.cc:1428] T 15caf455468a46839a03c2cc41554303 P f49d1db4eb4c4177b29a035b76a1846b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260924 16:06:05.686609 13739 tablet.cc:2369] T 15caf455468a46839a03c2cc41554303 P 98afcc80e1af45298f2af1a204b4dbca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:06:05.695501 14028 raft_consensus.cc:482] T 15caf455468a46839a03c2cc41554303 P 98afcc80e1af45298f2af1a204b4dbca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:06:05.695710 14028 raft_consensus.cc:504] T 15caf455468a46839a03c2cc41554303 P 98afcc80e1af45298f2af1a204b4dbca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb208e6d56b4d50a674793b25ee0ae9" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 45507 } } peers { permanent_uuid: "98afcc80e1af45298f2af1a204b4dbca" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 42323 } } peers { permanent_uuid: "f49d1db4eb4c4177b29a035b76a1846b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41789 } }
I20260924 16:06:05.697026 14028 leader_election.cc:288] T 15caf455468a46839a03c2cc41554303 P 98afcc80e1af45298f2af1a204b4dbca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbb208e6d56b4d50a674793b25ee0ae9 (127.25.109.2:45507), f49d1db4eb4c4177b29a035b76a1846b (127.25.109.3:41789)
I20260924 16:06:05.701323 14020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.697400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:40925 (local address 127.25.109.2:45507)
0924 16:06:05.697532 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:05.697538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:05.697654 (+ 116us) server_negotiation.cc:407] Connection header received
0924 16:06:05.697844 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:05.697847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:05.697915 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:05.698007 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:05.698380 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.699051 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.700274 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.700766 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.700861 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:05.700901 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:05.701114 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 16:06:05.701191 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 16:06:05.701341 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.697274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:45507 (local address 127.25.109.1:40925)
0924 16:06:05.697523 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:05.697548 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:05.697718 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:05.698036 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:05.698038 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:05.698057 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:05.698210 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:05.698221 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.699171 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.699174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:05.700145 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:05.700155 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.700881 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.700885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:05.700976 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:05.701005 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 16:06:05.701084 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 16:06:05.701156 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":97,"threads_started":1}
I20260924 16:06:05.701597 14032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.697690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.3:41789 (local address 127.25.109.1:35065)
0924 16:06:05.698028 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:05.698042 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:05.698163 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:05.698505 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:05.698508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:05.698526 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:05.698647 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:05.698654 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.699520 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.699522 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:05.700339 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:05.700349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.701149 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.701153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:05.701244 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:05.701269 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 16:06:05.701351 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 16:06:05.701425 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":74,"threads_started":1}
I20260924 16:06:05.701617 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.697802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.109.1:35065 (local address 127.25.109.3:41789)
0924 16:06:05.697945 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:05.697948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:05.698109 (+ 161us) server_negotiation.cc:407] Connection header received
0924 16:06:05.698340 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:05.698345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:05.698397 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:05.698464 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:05.698778 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.699403 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.700510 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.700983 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.701109 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:05.701151 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:05.701389 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 16:06:05.701478 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 16:06:05.702087 13826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15caf455468a46839a03c2cc41554303" candidate_uuid: "98afcc80e1af45298f2af1a204b4dbca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb208e6d56b4d50a674793b25ee0ae9" is_pre_election: true
I20260924 16:06:05.702263 13958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15caf455468a46839a03c2cc41554303" candidate_uuid: "98afcc80e1af45298f2af1a204b4dbca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f49d1db4eb4c4177b29a035b76a1846b" is_pre_election: true
I20260924 16:06:05.702451 13826 raft_consensus.cc:2454] T 15caf455468a46839a03c2cc41554303 P fbb208e6d56b4d50a674793b25ee0ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98afcc80e1af45298f2af1a204b4dbca in term 0.
I20260924 16:06:05.702574 13958 raft_consensus.cc:2454] T 15caf455468a46839a03c2cc41554303 P f49d1db4eb4c4177b29a035b76a1846b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98afcc80e1af45298f2af1a204b4dbca in term 0.
I20260924 16:06:05.702986 13625 leader_election.cc:302] T 15caf455468a46839a03c2cc41554303 P 98afcc80e1af45298f2af1a204b4dbca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98afcc80e1af45298f2af1a204b4dbca, fbb208e6d56b4d50a674793b25ee0ae9; no voters:
I20260924 16:06:05.703312 14028 raft_consensus.cc:2781] T 15caf455468a46839a03c2cc41554303 P 98afcc80e1af45298f2af1a204b4dbca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:06:05.703423 14028 raft_consensus.cc:482] T 15caf455468a46839a03c2cc41554303 P 98afcc80e1af45298f2af1a204b4dbca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:06:05.703508 14028 raft_consensus.cc:3037] T 15caf455468a46839a03c2cc41554303 P 98afcc80e1af45298f2af1a204b4dbca [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:05.704595 14028 raft_consensus.cc:504] T 15caf455468a46839a03c2cc41554303 P 98afcc80e1af45298f2af1a204b4dbca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb208e6d56b4d50a674793b25ee0ae9" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 45507 } } peers { permanent_uuid: "98afcc80e1af45298f2af1a204b4dbca" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 42323 } } peers { permanent_uuid: "f49d1db4eb4c4177b29a035b76a1846b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41789 } }
I20260924 16:06:05.704953 14028 leader_election.cc:288] T 15caf455468a46839a03c2cc41554303 P 98afcc80e1af45298f2af1a204b4dbca [CANDIDATE]: Term 1 election: Requested vote from peers fbb208e6d56b4d50a674793b25ee0ae9 (127.25.109.2:45507), f49d1db4eb4c4177b29a035b76a1846b (127.25.109.3:41789)
I20260924 16:06:05.705386 13826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15caf455468a46839a03c2cc41554303" candidate_uuid: "98afcc80e1af45298f2af1a204b4dbca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb208e6d56b4d50a674793b25ee0ae9"
I20260924 16:06:05.705431 13958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15caf455468a46839a03c2cc41554303" candidate_uuid: "98afcc80e1af45298f2af1a204b4dbca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f49d1db4eb4c4177b29a035b76a1846b"
I20260924 16:06:05.705556 13826 raft_consensus.cc:3037] T 15caf455468a46839a03c2cc41554303 P fbb208e6d56b4d50a674793b25ee0ae9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:05.705538 13958 raft_consensus.cc:3037] T 15caf455468a46839a03c2cc41554303 P f49d1db4eb4c4177b29a035b76a1846b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:05.706593 13958 raft_consensus.cc:2454] T 15caf455468a46839a03c2cc41554303 P f49d1db4eb4c4177b29a035b76a1846b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98afcc80e1af45298f2af1a204b4dbca in term 1.
I20260924 16:06:05.706681 13826 raft_consensus.cc:2454] T 15caf455468a46839a03c2cc41554303 P fbb208e6d56b4d50a674793b25ee0ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98afcc80e1af45298f2af1a204b4dbca in term 1.
I20260924 16:06:05.706933 13625 leader_election.cc:302] T 15caf455468a46839a03c2cc41554303 P 98afcc80e1af45298f2af1a204b4dbca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98afcc80e1af45298f2af1a204b4dbca, f49d1db4eb4c4177b29a035b76a1846b; no voters:
I20260924 16:06:05.707122 14028 raft_consensus.cc:2781] T 15caf455468a46839a03c2cc41554303 P 98afcc80e1af45298f2af1a204b4dbca [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:05.707378 14028 raft_consensus.cc:686] T 15caf455468a46839a03c2cc41554303 P 98afcc80e1af45298f2af1a204b4dbca [term 1 LEADER]: Becoming Leader. State: Replica: 98afcc80e1af45298f2af1a204b4dbca, State: Running, Role: LEADER
I20260924 16:06:05.707767 14028 consensus_queue.cc:237] T 15caf455468a46839a03c2cc41554303 P 98afcc80e1af45298f2af1a204b4dbca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb208e6d56b4d50a674793b25ee0ae9" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 45507 } } peers { permanent_uuid: "98afcc80e1af45298f2af1a204b4dbca" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 42323 } } peers { permanent_uuid: "f49d1db4eb4c4177b29a035b76a1846b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41789 } }
I20260924 16:06:05.711213 13545 catalog_manager.cc:5495] T 15caf455468a46839a03c2cc41554303 P 98afcc80e1af45298f2af1a204b4dbca reported cstate change: term changed from 0 to 1, leader changed from <none> to 98afcc80e1af45298f2af1a204b4dbca (127.25.109.1). New cstate: current_term: 1 leader_uuid: "98afcc80e1af45298f2af1a204b4dbca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb208e6d56b4d50a674793b25ee0ae9" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 45507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98afcc80e1af45298f2af1a204b4dbca" member_type: VOTER last_known_addr { host: "127.25.109.1" port: 42323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f49d1db4eb4c4177b29a035b76a1846b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41789 } health_report { overall_health: UNKNOWN } } }
I20260924 16:06:05.740604 14017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.736861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54860 (local address 127.25.109.1:42323)
0924 16:06:05.737018 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:05.737023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:05.737038 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:06:05.737215 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:05.737218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:05.737279 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:05.737398 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:06:05.737659 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.738171 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.739038 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:05.739267 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:05.739382 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:05.739706 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:06:05.740151 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:06:05.740299 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:05.740368 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 16:06:05.740448 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:06:05.747093 13958 raft_consensus.cc:1261] T 15caf455468a46839a03c2cc41554303 P f49d1db4eb4c4177b29a035b76a1846b [term 1 FOLLOWER]: Refusing update from remote peer 98afcc80e1af45298f2af1a204b4dbca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:06:05.747093 13826 raft_consensus.cc:1261] T 15caf455468a46839a03c2cc41554303 P fbb208e6d56b4d50a674793b25ee0ae9 [term 1 FOLLOWER]: Refusing update from remote peer 98afcc80e1af45298f2af1a204b4dbca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:06:05.747891 14033 consensus_queue.cc:1035] T 15caf455468a46839a03c2cc41554303 P 98afcc80e1af45298f2af1a204b4dbca [LEADER]: Connected to new peer: Peer: permanent_uuid: "f49d1db4eb4c4177b29a035b76a1846b" member_type: VOTER last_known_addr { host: "127.25.109.3" port: 41789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:06:05.748129 14028 consensus_queue.cc:1035] T 15caf455468a46839a03c2cc41554303 P 98afcc80e1af45298f2af1a204b4dbca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb208e6d56b4d50a674793b25ee0ae9" member_type: VOTER last_known_addr { host: "127.25.109.2" port: 45507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:06:05.759127 14037 mvcc.cc:205] Tried to move back new op lower bound from 7332929395691786240 to 7332929395542323200. Current Snapshot: MvccSnapshot[applied={T|T < 7332929395691786240 or (T in {7332929395691786240})}]
I20260924 16:06:05.759342 14038 mvcc.cc:205] Tried to move back new op lower bound from 7332929395691786240 to 7332929395542323200. Current Snapshot: MvccSnapshot[applied={T|T < 7332929395691786240}]
I20260924 16:06:05.764308 13547 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:32788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:06:05.764568 13547 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:32788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:06:05.766985 13547 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0f2a4c66c9247e49457a7884e87daf9: Sending DeleteTablet for 3 replicas of tablet 15caf455468a46839a03c2cc41554303
I20260924 16:06:05.767813 13671 tablet_service.cc:1514] Processing DeleteTablet for tablet 15caf455468a46839a03c2cc41554303 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:05 UTC) from {username='slave'} at 127.0.0.1:54854
I20260924 16:06:05.767875 13937 tablet_service.cc:1514] Processing DeleteTablet for tablet 15caf455468a46839a03c2cc41554303 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:05 UTC) from {username='slave'} at 127.0.0.1:44462
I20260924 16:06:05.768301 13804 tablet_service.cc:1514] Processing DeleteTablet for tablet 15caf455468a46839a03c2cc41554303 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:05 UTC) from {username='slave'} at 127.0.0.1:59616
I20260924 16:06:05.768679 14048 tablet_replica.cc:330] T 15caf455468a46839a03c2cc41554303 P f49d1db4eb4c4177b29a035b76a1846b: stopping tablet replica
I20260924 16:06:05.768833 14047 tablet_replica.cc:330] T 15caf455468a46839a03c2cc41554303 P 98afcc80e1af45298f2af1a204b4dbca: stopping tablet replica
I20260924 16:06:05.769100 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 13608
I20260924 16:06:05.769088 14048 raft_consensus.cc:2229] T 15caf455468a46839a03c2cc41554303 P f49d1db4eb4c4177b29a035b76a1846b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:06:05.769166 14047 raft_consensus.cc:2229] T 15caf455468a46839a03c2cc41554303 P 98afcc80e1af45298f2af1a204b4dbca [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:06:05.770661 14049 tablet_replica.cc:330] T 15caf455468a46839a03c2cc41554303 P fbb208e6d56b4d50a674793b25ee0ae9: stopping tablet replica
I20260924 16:06:05.770919 14049 raft_consensus.cc:2229] T 15caf455468a46839a03c2cc41554303 P fbb208e6d56b4d50a674793b25ee0ae9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:06:05.771160 14049 raft_consensus.cc:2258] T 15caf455468a46839a03c2cc41554303 P fbb208e6d56b4d50a674793b25ee0ae9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:05.771159 14048 raft_consensus.cc:2258] T 15caf455468a46839a03c2cc41554303 P f49d1db4eb4c4177b29a035b76a1846b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:05.772716 14048 ts_tablet_manager.cc:1905] T 15caf455468a46839a03c2cc41554303 P f49d1db4eb4c4177b29a035b76a1846b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:05.774893 14048 ts_tablet_manager.cc:1918] T 15caf455468a46839a03c2cc41554303 P f49d1db4eb4c4177b29a035b76a1846b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:06:05.774964 14048 log.cc:1196] T 15caf455468a46839a03c2cc41554303 P f49d1db4eb4c4177b29a035b76a1846b: Deleting WAL directory at /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265708917108-26036-0/minicluster-data/ts-2/wal/wals/15caf455468a46839a03c2cc41554303
I20260924 16:06:05.775178 14048 ts_tablet_manager.cc:1939] T 15caf455468a46839a03c2cc41554303 P f49d1db4eb4c4177b29a035b76a1846b: Deleting consensus metadata
I20260924 16:06:05.776309 14049 ts_tablet_manager.cc:1905] T 15caf455468a46839a03c2cc41554303 P fbb208e6d56b4d50a674793b25ee0ae9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 16:06:05.776443 13533 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.109.1:42323 (error 108)
I20260924 16:06:05.776544 13532 catalog_manager.cc:4841] TS f49d1db4eb4c4177b29a035b76a1846b (127.25.109.3:41789): tablet 15caf455468a46839a03c2cc41554303 (table test-table [id=74db5b07f9a34621ada1ed1b39ea7b70]) successfully deleted
I20260924 16:06:05.776657 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 13741
I20260924 16:06:05.778249 14018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.777716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.1:42323 (local address 127.0.0.1:54864)
0924 16:06:05.778063 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:05.778146 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.1:42323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211}
W20260924 16:06:05.778607 13533 catalog_manager.cc:4584] TS 98afcc80e1af45298f2af1a204b4dbca (127.25.109.1:42323): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 15caf455468a46839a03c2cc41554303: Network error: Client connection negotiation failed: client connection to 127.25.109.1:42323: connect: Connection refused (error 111)
I20260924 16:06:05.782985 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 13875
I20260924 16:06:05.784698 14018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.783267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.2:45507 (local address 127.0.0.1:59618)
0924 16:06:05.784566 (+ 1299us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:05.784628 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.2:45507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1194}
I20260924 16:06:05.789014 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 13514
2026-09-24T16:06:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1061 ms)
[----------] 6 tests from EncryptionPolicyTest (11088 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:06:05 dist-test-slave-n106 krb5kdc[14053](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:06:05 dist-test-slave-n106 krb5kdc[14053](info): set up 2 sockets
Sep 24 16:06:05 dist-test-slave-n106 krb5kdc[14053](info): commencing operation
krb5kdc: starting...
W20260924 16:06:06.841674 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:06:06 dist-test-slave-n106 krb5kdc[14053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:06:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:06:06.961596 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43909
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:07.071095 14066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:07.071400 14066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:07.071484 14066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:07.075538 14066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:07.075631 14066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:07.075675 14066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:07.075734 14066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:07.075774 14066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:07.081110 14066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43909
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:07.082146 14066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:07.083168 14066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:07.091667 14072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:07.091665 14074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:07.091729 14066 server_base.cc:1034] running on GCE node
W20260924 16:06:07.091662 14071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:07.092541 14066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:07.093639 14066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:07.094799 14066 hybrid_clock.cc:648] HybridClock initialized: now 1790265967094785 us; error 28 us; skew 500 ppm
Sep 24 16:06:07 dist-test-slave-n106 krb5kdc[14053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:07.097553 14066 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:06:07.098467 14066 webserver.cc:460] Webserver started at http://127.0.0.1:41021/ using document root <none> and password file <none>
I20260924 16:06:07.099026 14066 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:07.099116 14066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:07.099349 14066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:07.101485 14066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "5bf459dd87cb48fd9f4923196fb9f9ca"
format_stamp: "Formatted at 2026-09-24 16:06:07 on dist-test-slave-n106"
server_key: "d8973234026183e668cbb73de5bdb90b"
server_key_iv: "1781682c6c968bab7a67c7947febb2a9"
server_key_version: "encryptionkey@0"
I20260924 16:06:07.102035 14066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "5bf459dd87cb48fd9f4923196fb9f9ca"
format_stamp: "Formatted at 2026-09-24 16:06:07 on dist-test-slave-n106"
server_key: "d8973234026183e668cbb73de5bdb90b"
server_key_iv: "1781682c6c968bab7a67c7947febb2a9"
server_key_version: "encryptionkey@0"
I20260924 16:06:07.105736 14066 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:06:07.108127 14081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:07.109428 14066 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:06:07.109573 14066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "5bf459dd87cb48fd9f4923196fb9f9ca"
format_stamp: "Formatted at 2026-09-24 16:06:07 on dist-test-slave-n106"
server_key: "d8973234026183e668cbb73de5bdb90b"
server_key_iv: "1781682c6c968bab7a67c7947febb2a9"
server_key_version: "encryptionkey@0"
I20260924 16:06:07.109696 14066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:07.129458 14066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:07.132397 14066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:07.132678 14066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:07.140130 14066 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43909
I20260924 16:06:07.140161 14134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43909 every 8 connection(s)
I20260924 16:06:07.141176 14066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:07.144009 14135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:07.147738 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 14066
I20260924 16:06:07.147893 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:06:07.148185 26036 external_mini_cluster.cc:1428] Setting key f2bd181e284ba9cc42e19d17cf979321
I20260924 16:06:07.150303 14135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bf459dd87cb48fd9f4923196fb9f9ca: Bootstrap starting.
I20260924 16:06:07.153203 14135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bf459dd87cb48fd9f4923196fb9f9ca: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:07.154129 14135 log.cc:824] T 00000000000000000000000000000000 P 5bf459dd87cb48fd9f4923196fb9f9ca: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:07.156600 14135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bf459dd87cb48fd9f4923196fb9f9ca: No bootstrap required, opened a new log
Sep 24 16:06:07 dist-test-slave-n106 krb5kdc[14053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 16:06:07.160552 14135 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bf459dd87cb48fd9f4923196fb9f9ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf459dd87cb48fd9f4923196fb9f9ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43909 } }
I20260924 16:06:07.160835 14135 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bf459dd87cb48fd9f4923196fb9f9ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:07.160920 14135 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bf459dd87cb48fd9f4923196fb9f9ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf459dd87cb48fd9f4923196fb9f9ca, State: Initialized, Role: FOLLOWER
I20260924 16:06:07.161404 14135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bf459dd87cb48fd9f4923196fb9f9ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf459dd87cb48fd9f4923196fb9f9ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43909 } }
I20260924 16:06:07.161532 14135 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bf459dd87cb48fd9f4923196fb9f9ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:07.161630 14135 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bf459dd87cb48fd9f4923196fb9f9ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:07.161769 14135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bf459dd87cb48fd9f4923196fb9f9ca [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:07.163008 14135 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bf459dd87cb48fd9f4923196fb9f9ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf459dd87cb48fd9f4923196fb9f9ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43909 } }
I20260924 16:06:07.163444 14135 leader_election.cc:302] T 00000000000000000000000000000000 P 5bf459dd87cb48fd9f4923196fb9f9ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bf459dd87cb48fd9f4923196fb9f9ca; no voters:
I20260924 16:06:07.163849 14135 leader_election.cc:288] T 00000000000000000000000000000000 P 5bf459dd87cb48fd9f4923196fb9f9ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:06:07.163959 14140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bf459dd87cb48fd9f4923196fb9f9ca [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:07.164242 14140 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bf459dd87cb48fd9f4923196fb9f9ca [term 1 LEADER]: Becoming Leader. State: Replica: 5bf459dd87cb48fd9f4923196fb9f9ca, State: Running, Role: LEADER
I20260924 16:06:07.164657 14140 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bf459dd87cb48fd9f4923196fb9f9ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf459dd87cb48fd9f4923196fb9f9ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43909 } }
I20260924 16:06:07.165164 14138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:07.149745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43563 (local address 127.0.0.1:43909)
0924 16:06:07.150202 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:07.150222 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:07.150267 (+ 45us) server_negotiation.cc:407] Connection header received
0924 16:06:07.151110 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:07.151135 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:07.151597 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:07.151986 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:07.152746 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:07.154296 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:07.155393 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:07.156088 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:07.156259 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:07.159071 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:07.159077 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:07.159091 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:07.159127 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:07.161638 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:07.162275 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:07.162281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:07.162285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:07.162395 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:07.162758 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:07.162760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:07.162761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:07.163235 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:07.163415 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:07.163823 (+ 408us) server_negotiation.cc:299] Negotiation successful
0924 16:06:07.164082 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":121,"threads_started":1}
I20260924 16:06:07.165328 14135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bf459dd87cb48fd9f4923196fb9f9ca [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:07.167307 14141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bf459dd87cb48fd9f4923196fb9f9ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bf459dd87cb48fd9f4923196fb9f9ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf459dd87cb48fd9f4923196fb9f9ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43909 } } }
I20260924 16:06:07.167456 14141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bf459dd87cb48fd9f4923196fb9f9ca [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:07.167320 14142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bf459dd87cb48fd9f4923196fb9f9ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bf459dd87cb48fd9f4923196fb9f9ca. Latest consensus state: current_term: 1 leader_uuid: "5bf459dd87cb48fd9f4923196fb9f9ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf459dd87cb48fd9f4923196fb9f9ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43909 } } }
I20260924 16:06:07.168725 14142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bf459dd87cb48fd9f4923196fb9f9ca [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:07.169512 26036 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 16:06:07.171038 14154 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5bf459dd87cb48fd9f4923196fb9f9ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:06:07.171191 14154 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:06:07.171272 14155 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:07.172196 14155 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:07.178619 14155 catalog_manager.cc:1347] Generated new cluster ID: 0955f7452d804eec84de851353e907d0
I20260924 16:06:07.178711 14155 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:07.181313 14138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:07.171919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57499 (local address 127.0.0.1:43909)
0924 16:06:07.172049 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:07.172053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:07.172394 (+ 341us) server_negotiation.cc:407] Connection header received
0924 16:06:07.172579 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:07.172583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:07.172628 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:07.172722 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:07.173410 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:07.173943 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:07.175075 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:07.175452 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:07.175576 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:07.177361 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:07.177366 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:07.177369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:07.177395 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:07.179200 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:07.179764 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:07.179768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:07.179770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:07.179864 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:07.180211 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:07.180215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:07.180217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:07.180479 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:07.180633 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:07.181044 (+ 411us) server_negotiation.cc:299] Negotiation successful
0924 16:06:07.181144 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:06:07.206741 14155 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:07.207757 14155 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:07.215719 14155 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bf459dd87cb48fd9f4923196fb9f9ca: Generated new TSK 0
I20260924 16:06:07.216593 14155 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:06:07.229130 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 14066
W20260924 16:06:07.235445 14051 connection.cc:535] client connection to 127.0.0.1:43909 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43909): Connection reset by peer (error 104)
2026-09-24T16:06:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:06:07 dist-test-slave-n106 krb5kdc[14166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:06:07 dist-test-slave-n106 krb5kdc[14166](info): set up 2 sockets
Sep 24 16:06:07 dist-test-slave-n106 krb5kdc[14166](info): commencing operation
krb5kdc: starting...
W20260924 16:06:08.273754 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:06:08 dist-test-slave-n106 krb5kdc[14166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:06:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:06:08.395635 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36759
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:08.513624 14179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:08.513891 14179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:08.513934 14179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:08.517939 14179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:08.518021 14179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:08.518043 14179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:08.518060 14179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:08.518075 14179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:08.523491 14179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36759
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:08.524526 14179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:08.525498 14179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:08.534144 14184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:08.534235 14185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:08.534147 14187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:08.534205 14179 server_base.cc:1034] running on GCE node
I20260924 16:06:08.534920 14179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:08.536072 14179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:08.537268 14179 hybrid_clock.cc:648] HybridClock initialized: now 1790265968537224 us; error 57 us; skew 500 ppm
Sep 24 16:06:08 dist-test-slave-n106 krb5kdc[14166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:08.540071 14179 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:06:08.541066 14179 webserver.cc:460] Webserver started at http://127.0.0.1:43809/ using document root <none> and password file <none>
I20260924 16:06:08.541649 14179 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:08.541744 14179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:08.541988 14179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:08.544076 14179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "b4c1c28104ec4ede9fdb353c62a3ea9a"
format_stamp: "Formatted at 2026-09-24 16:06:08 on dist-test-slave-n106"
server_key: "57c03f8a40e23949e56c5b079bbcf8e6"
server_key_iv: "58c35f0d2a21d62e496db1d162177a32"
server_key_version: "encryptionkey@0"
I20260924 16:06:08.544679 14179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "b4c1c28104ec4ede9fdb353c62a3ea9a"
format_stamp: "Formatted at 2026-09-24 16:06:08 on dist-test-slave-n106"
server_key: "57c03f8a40e23949e56c5b079bbcf8e6"
server_key_iv: "58c35f0d2a21d62e496db1d162177a32"
server_key_version: "encryptionkey@0"
I20260924 16:06:08.548377 14179 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:06:08.550814 14195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:08.552090 14179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:06:08.552240 14179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "b4c1c28104ec4ede9fdb353c62a3ea9a"
format_stamp: "Formatted at 2026-09-24 16:06:08 on dist-test-slave-n106"
server_key: "57c03f8a40e23949e56c5b079bbcf8e6"
server_key_iv: "58c35f0d2a21d62e496db1d162177a32"
server_key_version: "encryptionkey@0"
I20260924 16:06:08.552358 14179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:08.582018 14179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:08.585217 14179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:08.585439 14179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:08.592724 14179 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36759
I20260924 16:06:08.592816 14247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36759 every 8 connection(s)
I20260924 16:06:08.593748 14179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:08.596552 14248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:08.602155 14248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4c1c28104ec4ede9fdb353c62a3ea9a: Bootstrap starting.
I20260924 16:06:08.602788 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 14179
I20260924 16:06:08.602946 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:06:08.603250 26036 external_mini_cluster.cc:1428] Setting key 7dea15a06ac81363cf46712db196d2cc
I20260924 16:06:08.605063 14248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4c1c28104ec4ede9fdb353c62a3ea9a: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:08.605842 14248 log.cc:824] T 00000000000000000000000000000000 P b4c1c28104ec4ede9fdb353c62a3ea9a: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:08.607779 14248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4c1c28104ec4ede9fdb353c62a3ea9a: No bootstrap required, opened a new log
I20260924 16:06:08.611604 14248 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4c1c28104ec4ede9fdb353c62a3ea9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1c28104ec4ede9fdb353c62a3ea9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36759 } }
I20260924 16:06:08.611857 14248 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4c1c28104ec4ede9fdb353c62a3ea9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:08.611954 14248 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4c1c28104ec4ede9fdb353c62a3ea9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c1c28104ec4ede9fdb353c62a3ea9a, State: Initialized, Role: FOLLOWER
I20260924 16:06:08.612387 14251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:08.604580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56173 (local address 127.0.0.1:36759)
0924 16:06:08.604998 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:08.605013 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:08.605050 (+ 37us) server_negotiation.cc:407] Connection header received
0924 16:06:08.605716 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:08.605733 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:08.606132 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:08.606477 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:08.606900 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:08.607859 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:08.608877 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:08.609362 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:08.609532 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:08.610524 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:08.610529 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:08.610542 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:08.610921 (+ 379us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:08.610987 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:08.611011 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:08.611154 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:08.611338 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 16:06:08.611480 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":123,"threads_started":1}
I20260924 16:06:08.612393 14248 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4c1c28104ec4ede9fdb353c62a3ea9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1c28104ec4ede9fdb353c62a3ea9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36759 } }
I20260924 16:06:08.612721 14248 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4c1c28104ec4ede9fdb353c62a3ea9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:08.612795 14248 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4c1c28104ec4ede9fdb353c62a3ea9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:08.612895 14248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4c1c28104ec4ede9fdb353c62a3ea9a [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:08.613922 14248 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4c1c28104ec4ede9fdb353c62a3ea9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1c28104ec4ede9fdb353c62a3ea9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36759 } }
I20260924 16:06:08.614272 14248 leader_election.cc:302] T 00000000000000000000000000000000 P b4c1c28104ec4ede9fdb353c62a3ea9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4c1c28104ec4ede9fdb353c62a3ea9a; no voters:
I20260924 16:06:08.614653 14248 leader_election.cc:288] T 00000000000000000000000000000000 P b4c1c28104ec4ede9fdb353c62a3ea9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:06:08.614748 14253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4c1c28104ec4ede9fdb353c62a3ea9a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:08.614965 14253 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4c1c28104ec4ede9fdb353c62a3ea9a [term 1 LEADER]: Becoming Leader. State: Replica: b4c1c28104ec4ede9fdb353c62a3ea9a, State: Running, Role: LEADER
I20260924 16:06:08.615338 14253 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4c1c28104ec4ede9fdb353c62a3ea9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1c28104ec4ede9fdb353c62a3ea9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36759 } }
I20260924 16:06:08.615614 26036 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:06:08.616039 14248 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4c1c28104ec4ede9fdb353c62a3ea9a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:08.617239 14254 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4c1c28104ec4ede9fdb353c62a3ea9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4c1c28104ec4ede9fdb353c62a3ea9a. Latest consensus state: current_term: 1 leader_uuid: "b4c1c28104ec4ede9fdb353c62a3ea9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1c28104ec4ede9fdb353c62a3ea9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36759 } } }
I20260924 16:06:08.617239 14255 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4c1c28104ec4ede9fdb353c62a3ea9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4c1c28104ec4ede9fdb353c62a3ea9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1c28104ec4ede9fdb353c62a3ea9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36759 } } }
I20260924 16:06:08.617393 14255 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4c1c28104ec4ede9fdb353c62a3ea9a [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:08.617393 14254 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4c1c28104ec4ede9fdb353c62a3ea9a [sys.catalog]: This master's current role is: LEADER
W20260924 16:06:08.621206 14273 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b4c1c28104ec4ede9fdb353c62a3ea9a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:06:08.621296 14273 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:06:08.621362 14266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:08.622264 14266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:08.622885 14251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:08.617944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46385 (local address 127.0.0.1:36759)
0924 16:06:08.618055 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:08.618059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:08.618267 (+ 208us) server_negotiation.cc:407] Connection header received
0924 16:06:08.618449 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:08.618452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:08.618508 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:08.618555 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:08.618961 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:08.619625 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:08.620779 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:08.621124 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:08.621197 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:08.622323 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:08.622327 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:08.622330 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:08.622343 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:08.622369 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:08.622382 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:08.622443 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:08.622707 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 16:06:08.622747 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 16:06:08.627848 14266 catalog_manager.cc:1347] Generated new cluster ID: c5c139669b8748478e462110bb88a126
I20260924 16:06:08.627918 14266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:08.651772 14266 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:08.653225 14266 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:08.664582 14266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4c1c28104ec4ede9fdb353c62a3ea9a: Generated new TSK 0
I20260924 16:06:08.665509 14266 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:06:08.677222 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 14179
W20260924 16:06:08.683336 14164 connection.cc:535] client connection to 127.0.0.1:36759 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36759): Connection reset by peer (error 104)
2026-09-24T16:06:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1447 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:06:08 dist-test-slave-n106 krb5kdc[14279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:06:08 dist-test-slave-n106 krb5kdc[14279](info): set up 2 sockets
Sep 24 16:06:08 dist-test-slave-n106 krb5kdc[14279](info): commencing operation
krb5kdc: starting...
W20260924 16:06:09.733605 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:06:09 dist-test-slave-n106 krb5kdc[14279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:06:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:06:09.859267 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37163
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:09.970072 14292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:09.970345 14292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:09.970386 14292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:09.974383 14292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:09.974439 14292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:09.974458 14292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:09.974475 14292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:09.974490 14292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:09.979591 14292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37163
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:09.980666 14292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:09.981650 14292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:09.990049 14298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:09.990084 14300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:09.990120 14297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:09.990211 14292 server_base.cc:1034] running on GCE node
I20260924 16:06:09.990891 14292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:09.991967 14292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:09.993160 14292 hybrid_clock.cc:648] HybridClock initialized: now 1790265969993136 us; error 42 us; skew 500 ppm
Sep 24 16:06:10 dist-test-slave-n106 krb5kdc[14279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:10.008378 14292 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:06:10.009443 14292 webserver.cc:460] Webserver started at http://127.0.0.1:38261/ using document root <none> and password file <none>
I20260924 16:06:10.010033 14292 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:10.010133 14292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:10.010375 14292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:10.012547 14292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "f0e6439dc95b4711947b57f453740f6d"
format_stamp: "Formatted at 2026-09-24 16:06:10 on dist-test-slave-n106"
server_key: "a6a636abd2b1544153387167533c4f97"
server_key_iv: "227df8dfc6f482b2a43dcdaec78072d1"
server_key_version: "encryptionkey@0"
I20260924 16:06:10.013126 14292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "f0e6439dc95b4711947b57f453740f6d"
format_stamp: "Formatted at 2026-09-24 16:06:10 on dist-test-slave-n106"
server_key: "a6a636abd2b1544153387167533c4f97"
server_key_iv: "227df8dfc6f482b2a43dcdaec78072d1"
server_key_version: "encryptionkey@0"
I20260924 16:06:10.016848 14292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:06:10.019232 14307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:10.020547 14292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:06:10.020682 14292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "f0e6439dc95b4711947b57f453740f6d"
format_stamp: "Formatted at 2026-09-24 16:06:10 on dist-test-slave-n106"
server_key: "a6a636abd2b1544153387167533c4f97"
server_key_iv: "227df8dfc6f482b2a43dcdaec78072d1"
server_key_version: "encryptionkey@0"
I20260924 16:06:10.020792 14292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:10.025363 14292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:10.028630 14292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:10.028816 14292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:10.036145 14292 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37163
I20260924 16:06:10.036160 14360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37163 every 8 connection(s)
I20260924 16:06:10.037144 14292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:10.039889 14361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:10.045552 14361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0e6439dc95b4711947b57f453740f6d: Bootstrap starting.
I20260924 16:06:10.045827 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 14292
I20260924 16:06:10.045984 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:06:10.046305 26036 external_mini_cluster.cc:1428] Setting key 8c8c1c81f89b7e6b79125b4d791665bd
I20260924 16:06:10.048326 14361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0e6439dc95b4711947b57f453740f6d: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:10.049096 14361 log.cc:824] T 00000000000000000000000000000000 P f0e6439dc95b4711947b57f453740f6d: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:10.051326 14361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0e6439dc95b4711947b57f453740f6d: No bootstrap required, opened a new log
I20260924 16:06:10.051730 14364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:10.047671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59045 (local address 127.0.0.1:37163)
0924 16:06:10.048097 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:10.048122 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:10.048157 (+ 35us) server_negotiation.cc:407] Connection header received
0924 16:06:10.048854 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:10.048873 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:10.049166 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:10.049534 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:10.050338 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:10.050344 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:10.050359 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:10.050371 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:10.050430 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:10.050455 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:10.050568 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:10.050676 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 16:06:10.050826 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":119,"threads_started":1}
I20260924 16:06:10.054500 26036 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:06:10.055397 14361 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0e6439dc95b4711947b57f453740f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e6439dc95b4711947b57f453740f6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37163 } }
I20260924 16:06:10.055642 14361 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0e6439dc95b4711947b57f453740f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:10.055707 14361 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0e6439dc95b4711947b57f453740f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e6439dc95b4711947b57f453740f6d, State: Initialized, Role: FOLLOWER
I20260924 16:06:10.056150 14361 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0e6439dc95b4711947b57f453740f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e6439dc95b4711947b57f453740f6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37163 } }
I20260924 16:06:10.056303 14361 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0e6439dc95b4711947b57f453740f6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:10.056387 14361 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0e6439dc95b4711947b57f453740f6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:10.056512 14361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0e6439dc95b4711947b57f453740f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:10.057631 14361 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0e6439dc95b4711947b57f453740f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e6439dc95b4711947b57f453740f6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37163 } }
I20260924 16:06:10.057996 14361 leader_election.cc:302] T 00000000000000000000000000000000 P f0e6439dc95b4711947b57f453740f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0e6439dc95b4711947b57f453740f6d; no voters:
I20260924 16:06:10.058240 14364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:10.056610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41261 (local address 127.0.0.1:37163)
0924 16:06:10.056731 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:10.056738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:10.056956 (+ 218us) server_negotiation.cc:407] Connection header received
0924 16:06:10.057034 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:10.057037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:10.057071 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:10.057161 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:10.057893 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:10.057896 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:10.057899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:10.057910 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:10.057938 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:10.057950 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:10.058046 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:10.058084 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 16:06:10.058132 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 16:06:10.058353 14361 leader_election.cc:288] T 00000000000000000000000000000000 P f0e6439dc95b4711947b57f453740f6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:06:10.058488 14371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0e6439dc95b4711947b57f453740f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:10.058724 14371 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0e6439dc95b4711947b57f453740f6d [term 1 LEADER]: Becoming Leader. State: Replica: f0e6439dc95b4711947b57f453740f6d, State: Running, Role: LEADER
I20260924 16:06:10.059024 14371 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0e6439dc95b4711947b57f453740f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e6439dc95b4711947b57f453740f6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37163 } }
I20260924 16:06:10.059681 14361 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0e6439dc95b4711947b57f453740f6d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:10.060839 14373 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0e6439dc95b4711947b57f453740f6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0e6439dc95b4711947b57f453740f6d. Latest consensus state: current_term: 1 leader_uuid: "f0e6439dc95b4711947b57f453740f6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e6439dc95b4711947b57f453740f6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37163 } } }
I20260924 16:06:10.060889 14372 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0e6439dc95b4711947b57f453740f6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0e6439dc95b4711947b57f453740f6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e6439dc95b4711947b57f453740f6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37163 } } }
I20260924 16:06:10.060976 14373 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0e6439dc95b4711947b57f453740f6d [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:10.060999 14372 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0e6439dc95b4711947b57f453740f6d [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:10.061424 14380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:10.065240 14380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:10.071440 14380 catalog_manager.cc:1347] Generated new cluster ID: 90950fa915cb48ef86d4eec5074f968e
I20260924 16:06:10.071517 14380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:10.089422 14380 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:10.089775 14380 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:10.096271 14380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0e6439dc95b4711947b57f453740f6d: Generated new TSK 0
I20260924 16:06:10.097004 14380 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:06:10.106523 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 14292
2026-09-24T16:06:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:06:10 dist-test-slave-n106 krb5kdc[14392](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:06:10 dist-test-slave-n106 krb5kdc[14392](info): set up 2 sockets
Sep 24 16:06:10 dist-test-slave-n106 krb5kdc[14392](info): commencing operation
krb5kdc: starting...
W20260924 16:06:11.145566 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:06:11 dist-test-slave-n106 krb5kdc[14392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:06:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:06:11.267498 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42253
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:11.377452 14405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:11.377718 14405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:11.377797 14405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:11.381843 14405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:11.381898 14405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:11.381918 14405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:11.381934 14405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:11.381973 14405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:11.387141 14405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:11.388190 14405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:11.389278 14405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:11.397698 14411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:11.397680 14410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:11.397804 14405 server_base.cc:1034] running on GCE node
W20260924 16:06:11.397679 14413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:11.398437 14405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:11.399528 14405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:11.400697 14405 hybrid_clock.cc:648] HybridClock initialized: now 1790265971400665 us; error 42 us; skew 500 ppm
Sep 24 16:06:11 dist-test-slave-n106 krb5kdc[14392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:11.403465 14405 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:06:11.404318 14405 webserver.cc:460] Webserver started at http://127.0.0.1:37331/ using document root <none> and password file <none>
I20260924 16:06:11.404911 14405 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:11.404971 14405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:11.405207 14405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:11.407207 14405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "0edf2da2bccf45d0becace089102df90"
format_stamp: "Formatted at 2026-09-24 16:06:11 on dist-test-slave-n106"
server_key: "793c49300b5a326b1e8a0163eac9f5b1"
server_key_iv: "3616c83c20191c2c76c5e564f83b3f11"
server_key_version: "encryptionkey@0"
I20260924 16:06:11.407756 14405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "0edf2da2bccf45d0becace089102df90"
format_stamp: "Formatted at 2026-09-24 16:06:11 on dist-test-slave-n106"
server_key: "793c49300b5a326b1e8a0163eac9f5b1"
server_key_iv: "3616c83c20191c2c76c5e564f83b3f11"
server_key_version: "encryptionkey@0"
I20260924 16:06:11.411475 14405 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 16:06:11.413841 14421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:11.414983 14405 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:06:11.415128 14405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "0edf2da2bccf45d0becace089102df90"
format_stamp: "Formatted at 2026-09-24 16:06:11 on dist-test-slave-n106"
server_key: "793c49300b5a326b1e8a0163eac9f5b1"
server_key_iv: "3616c83c20191c2c76c5e564f83b3f11"
server_key_version: "encryptionkey@0"
I20260924 16:06:11.415223 14405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:11.437394 14405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:11.440678 14405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:11.440877 14405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:11.447844 14405 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42253
I20260924 16:06:11.447866 14475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42253 every 8 connection(s)
I20260924 16:06:11.448817 14405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:11.451553 14476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:11.453891 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 14405
I20260924 16:06:11.454000 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:06:11.454284 26036 external_mini_cluster.cc:1428] Setting key 5316631a2170184134a02b49c0e3df9b
I20260924 16:06:11.457974 14476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0edf2da2bccf45d0becace089102df90: Bootstrap starting.
I20260924 16:06:11.460626 14476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0edf2da2bccf45d0becace089102df90: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:11.461539 14476 log.cc:824] T 00000000000000000000000000000000 P 0edf2da2bccf45d0becace089102df90: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:11.463553 14476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0edf2da2bccf45d0becace089102df90: No bootstrap required, opened a new log
Sep 24 16:06:11 dist-test-slave-n106 krb5kdc[14392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 16:06:11.467401 14476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0edf2da2bccf45d0becace089102df90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0edf2da2bccf45d0becace089102df90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42253 } }
I20260924 16:06:11.467650 14476 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0edf2da2bccf45d0becace089102df90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:11.467741 14476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0edf2da2bccf45d0becace089102df90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0edf2da2bccf45d0becace089102df90, State: Initialized, Role: FOLLOWER
I20260924 16:06:11.468241 14476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0edf2da2bccf45d0becace089102df90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0edf2da2bccf45d0becace089102df90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42253 } }
I20260924 16:06:11.468379 14476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0edf2da2bccf45d0becace089102df90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:11.468520 14476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0edf2da2bccf45d0becace089102df90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:11.468633 14476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0edf2da2bccf45d0becace089102df90 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:11.469290 14479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:11.455665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51887 (local address 127.0.0.1:42253)
0924 16:06:11.456077 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:11.456096 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:11.456138 (+ 42us) server_negotiation.cc:407] Connection header received
0924 16:06:11.456990 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:11.457016 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:11.457433 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:11.457816 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:11.458538 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:11.460001 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:11.460941 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:11.461628 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:11.461798 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:11.464509 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:11.464514 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:11.464527 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:11.464554 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:11.466246 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:11.466739 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:11.466743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:11.466747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:11.466857 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:11.467151 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:11.467154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:11.467155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:11.467512 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:11.467729 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:11.468056 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 16:06:11.468395 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":114,"threads_started":1}
I20260924 16:06:11.469534 14476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0edf2da2bccf45d0becace089102df90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0edf2da2bccf45d0becace089102df90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42253 } }
I20260924 16:06:11.469889 14476 leader_election.cc:302] T 00000000000000000000000000000000 P 0edf2da2bccf45d0becace089102df90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0edf2da2bccf45d0becace089102df90; no voters:
I20260924 16:06:11.470201 14476 leader_election.cc:288] T 00000000000000000000000000000000 P 0edf2da2bccf45d0becace089102df90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:06:11.470311 14481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0edf2da2bccf45d0becace089102df90 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:11.470526 14481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0edf2da2bccf45d0becace089102df90 [term 1 LEADER]: Becoming Leader. State: Replica: 0edf2da2bccf45d0becace089102df90, State: Running, Role: LEADER
I20260924 16:06:11.470894 14481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0edf2da2bccf45d0becace089102df90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0edf2da2bccf45d0becace089102df90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42253 } }
I20260924 16:06:11.471573 14476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0edf2da2bccf45d0becace089102df90 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:11.472102 14481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0edf2da2bccf45d0becace089102df90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0edf2da2bccf45d0becace089102df90. Latest consensus state: current_term: 1 leader_uuid: "0edf2da2bccf45d0becace089102df90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0edf2da2bccf45d0becace089102df90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42253 } } }
I20260924 16:06:11.472297 14481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0edf2da2bccf45d0becace089102df90 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:11.472509 26036 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:06:11.472585 14482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0edf2da2bccf45d0becace089102df90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0edf2da2bccf45d0becace089102df90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0edf2da2bccf45d0becace089102df90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42253 } } }
I20260924 16:06:11.472678 14482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0edf2da2bccf45d0becace089102df90 [sys.catalog]: This master's current role is: LEADER
W20260924 16:06:11.477013 14498 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0edf2da2bccf45d0becace089102df90: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:06:11.477106 14498 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:06:11.477218 14489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:11.477880 14489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:11.483119 14489 catalog_manager.cc:1347] Generated new cluster ID: 74cdbc6467ab4d909451104679589c77
I20260924 16:06:11.483172 14489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:11.484285 14479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:11.474925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35465 (local address 127.0.0.1:42253)
0924 16:06:11.475047 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:11.475051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:11.475276 (+ 225us) server_negotiation.cc:407] Connection header received
0924 16:06:11.475409 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:11.475411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:11.475461 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:11.475539 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:11.476323 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:11.477092 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:11.478267 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:11.478634 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:11.478738 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:11.480400 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:11.480438 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:11.480441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:11.480466 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:11.482200 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:11.482678 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:11.482683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:11.482688 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:11.482773 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:11.483086 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:11.483089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:11.483091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:11.483297 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:11.483431 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:11.484031 (+ 600us) server_negotiation.cc:299] Negotiation successful
0924 16:06:11.484139 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 16:06:11.503631 14489 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:11.504565 14489 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:11.508729 14489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0edf2da2bccf45d0becace089102df90: Generated new TSK 0
I20260924 16:06:11.509392 14489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:06:11.520368 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 14405
2026-09-24T16:06:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1412 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:06:11 dist-test-slave-n106 krb5kdc[14509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:06:11 dist-test-slave-n106 krb5kdc[14509](info): set up 2 sockets
Sep 24 16:06:11 dist-test-slave-n106 krb5kdc[14509](info): commencing operation
krb5kdc: starting...
W20260924 16:06:12.561628 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:06:12 dist-test-slave-n106 krb5kdc[14509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:06:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:06:12.685534 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40507
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:12.797142 14523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:12.797401 14523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:12.797449 14523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:12.801855 14523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:12.801909 14523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:12.801934 14523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:12.801952 14523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:12.801967 14523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:12.807305 14523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:12.808336 14523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:12.809384 14523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:12.818074 14529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:12.818063 14528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:12.818192 14523 server_base.cc:1034] running on GCE node
W20260924 16:06:12.818066 14531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:12.818862 14523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:12.819867 14523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:12.821022 14523 hybrid_clock.cc:648] HybridClock initialized: now 1790265972821003 us; error 30 us; skew 500 ppm
Sep 24 16:06:12 dist-test-slave-n106 krb5kdc[14509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:12.823599 14523 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:06:12.824440 14523 webserver.cc:460] Webserver started at http://127.0.0.1:41781/ using document root <none> and password file <none>
I20260924 16:06:12.824968 14523 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:12.825023 14523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:12.825181 14523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:12.827091 14523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "a94db7ed3f8e401181a47b2f8fb68dbc"
format_stamp: "Formatted at 2026-09-24 16:06:12 on dist-test-slave-n106"
server_key: "79a668c93e1f2a7d681947e9b5940a71"
server_key_iv: "2b1d32fe14f92d6983888a8f571fa615"
server_key_version: "encryptionkey@0"
I20260924 16:06:12.827579 14523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "a94db7ed3f8e401181a47b2f8fb68dbc"
format_stamp: "Formatted at 2026-09-24 16:06:12 on dist-test-slave-n106"
server_key: "79a668c93e1f2a7d681947e9b5940a71"
server_key_iv: "2b1d32fe14f92d6983888a8f571fa615"
server_key_version: "encryptionkey@0"
I20260924 16:06:12.831130 14523 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 16:06:12.833477 14538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:12.834669 14523 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:06:12.834765 14523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "a94db7ed3f8e401181a47b2f8fb68dbc"
format_stamp: "Formatted at 2026-09-24 16:06:12 on dist-test-slave-n106"
server_key: "79a668c93e1f2a7d681947e9b5940a71"
server_key_iv: "2b1d32fe14f92d6983888a8f571fa615"
server_key_version: "encryptionkey@0"
I20260924 16:06:12.834846 14523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:12.872579 14523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:12.875384 14523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:12.875542 14523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:12.882627 14523 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40507
I20260924 16:06:12.882644 14590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40507 every 8 connection(s)
I20260924 16:06:12.883600 14523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:12.886356 14591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:12.891780 14591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a94db7ed3f8e401181a47b2f8fb68dbc: Bootstrap starting.
I20260924 16:06:12.892115 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 14523
I20260924 16:06:12.892263 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:06:12.892596 26036 external_mini_cluster.cc:1428] Setting key 538c42e31435005742336dc39fbe205b
I20260924 16:06:12.894454 14591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a94db7ed3f8e401181a47b2f8fb68dbc: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:12.895153 14591 log.cc:824] T 00000000000000000000000000000000 P a94db7ed3f8e401181a47b2f8fb68dbc: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:12.896991 14591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a94db7ed3f8e401181a47b2f8fb68dbc: No bootstrap required, opened a new log
I20260924 16:06:12.900558 14591 raft_consensus.cc:348] T 00000000000000000000000000000000 P a94db7ed3f8e401181a47b2f8fb68dbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a94db7ed3f8e401181a47b2f8fb68dbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40507 } }
I20260924 16:06:12.900856 14591 raft_consensus.cc:374] T 00000000000000000000000000000000 P a94db7ed3f8e401181a47b2f8fb68dbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:12.900949 14591 raft_consensus.cc:729] T 00000000000000000000000000000000 P a94db7ed3f8e401181a47b2f8fb68dbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a94db7ed3f8e401181a47b2f8fb68dbc, State: Initialized, Role: FOLLOWER
I20260924 16:06:12.901429 14591 consensus_queue.cc:260] T 00000000000000000000000000000000 P a94db7ed3f8e401181a47b2f8fb68dbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a94db7ed3f8e401181a47b2f8fb68dbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40507 } }
I20260924 16:06:12.901561 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:12.893914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37103 (local address 127.0.0.1:40507)
0924 16:06:12.894330 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:12.894345 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:12.894379 (+ 34us) server_negotiation.cc:407] Connection header received
0924 16:06:12.895045 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:12.895063 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:12.895444 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:12.895772 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:12.896181 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:12.897263 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:12.898170 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:12.898657 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:12.898833 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:12.899748 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:12.899753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:12.899766 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:12.900142 (+ 376us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:12.900194 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:12.900216 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:12.900327 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:12.900491 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 16:06:12.900705 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":126,"threads_started":1}
I20260924 16:06:12.901552 14591 raft_consensus.cc:388] T 00000000000000000000000000000000 P a94db7ed3f8e401181a47b2f8fb68dbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:12.901700 14591 raft_consensus.cc:482] T 00000000000000000000000000000000 P a94db7ed3f8e401181a47b2f8fb68dbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:12.901798 14591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a94db7ed3f8e401181a47b2f8fb68dbc [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:12.902745 14591 raft_consensus.cc:504] T 00000000000000000000000000000000 P a94db7ed3f8e401181a47b2f8fb68dbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a94db7ed3f8e401181a47b2f8fb68dbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40507 } }
I20260924 16:06:12.903100 14591 leader_election.cc:302] T 00000000000000000000000000000000 P a94db7ed3f8e401181a47b2f8fb68dbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a94db7ed3f8e401181a47b2f8fb68dbc; no voters:
I20260924 16:06:12.903482 14591 leader_election.cc:288] T 00000000000000000000000000000000 P a94db7ed3f8e401181a47b2f8fb68dbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:06:12.903575 14596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a94db7ed3f8e401181a47b2f8fb68dbc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:12.903861 14596 raft_consensus.cc:686] T 00000000000000000000000000000000 P a94db7ed3f8e401181a47b2f8fb68dbc [term 1 LEADER]: Becoming Leader. State: Replica: a94db7ed3f8e401181a47b2f8fb68dbc, State: Running, Role: LEADER
I20260924 16:06:12.904266 14596 consensus_queue.cc:237] T 00000000000000000000000000000000 P a94db7ed3f8e401181a47b2f8fb68dbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a94db7ed3f8e401181a47b2f8fb68dbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40507 } }
I20260924 16:06:12.904533 26036 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:06:12.904794 14591 sys_catalog.cc:564] T 00000000000000000000000000000000 P a94db7ed3f8e401181a47b2f8fb68dbc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:12.905879 14598 sys_catalog.cc:455] T 00000000000000000000000000000000 P a94db7ed3f8e401181a47b2f8fb68dbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader a94db7ed3f8e401181a47b2f8fb68dbc. Latest consensus state: current_term: 1 leader_uuid: "a94db7ed3f8e401181a47b2f8fb68dbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a94db7ed3f8e401181a47b2f8fb68dbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40507 } } }
I20260924 16:06:12.906018 14598 sys_catalog.cc:458] T 00000000000000000000000000000000 P a94db7ed3f8e401181a47b2f8fb68dbc [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:12.906345 14597 sys_catalog.cc:455] T 00000000000000000000000000000000 P a94db7ed3f8e401181a47b2f8fb68dbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a94db7ed3f8e401181a47b2f8fb68dbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a94db7ed3f8e401181a47b2f8fb68dbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40507 } } }
I20260924 16:06:12.906486 14597 sys_catalog.cc:458] T 00000000000000000000000000000000 P a94db7ed3f8e401181a47b2f8fb68dbc [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:12.907420 14609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:12.910284 14609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:12.911496 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:12.906924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32939 (local address 127.0.0.1:40507)
0924 16:06:12.907048 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:12.907052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:12.907222 (+ 170us) server_negotiation.cc:407] Connection header received
0924 16:06:12.907409 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:12.907412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:12.907469 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:12.907520 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:12.907903 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:12.908648 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:12.909772 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:12.910081 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:12.910188 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:12.911051 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:12.911055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:12.911058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:12.911071 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:12.911099 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:12.911113 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:12.911193 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:12.911317 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 16:06:12.911366 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:06:12.915839 14609 catalog_manager.cc:1347] Generated new cluster ID: c4c00fb864b8404da855361f9c3691bd
I20260924 16:06:12.915920 14609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:12.931939 14609 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:12.933266 14609 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:12.942322 14609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a94db7ed3f8e401181a47b2f8fb68dbc: Generated new TSK 0
I20260924 16:06:12.942965 14609 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:06:12.946578 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 14523
2026-09-24T16:06:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1426 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:06:13 dist-test-slave-n106 krb5kdc[14623](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:06:13 dist-test-slave-n106 krb5kdc[14623](info): set up 2 sockets
Sep 24 16:06:13 dist-test-slave-n106 krb5kdc[14623](info): commencing operation
krb5kdc: starting...
W20260924 16:06:13.993649 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:06:14 dist-test-slave-n106 krb5kdc[14623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:06:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:06:14.113502 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34843
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:14.223956 14637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:14.224200 14637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:14.224246 14637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:14.228192 14637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:14.228243 14637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:14.228263 14637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:14.228279 14637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:14.228300 14637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:14.233628 14637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:14.234596 14637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:14.235630 14637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:14.243841 14643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:14.243853 14645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:14.243853 14642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:14.244136 14637 server_base.cc:1034] running on GCE node
I20260924 16:06:14.244838 14637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:14.245930 14637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:14.247130 14637 hybrid_clock.cc:648] HybridClock initialized: now 1790265974247118 us; error 30 us; skew 500 ppm
Sep 24 16:06:14 dist-test-slave-n106 krb5kdc[14623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:14.250052 14637 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:06:14.250937 14637 webserver.cc:460] Webserver started at http://127.0.0.1:37793/ using document root <none> and password file <none>
I20260924 16:06:14.251507 14637 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:14.251600 14637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:14.251840 14637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:14.253948 14637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "c8f5c6a78d2d4607b1f8ef0cb09c1429"
format_stamp: "Formatted at 2026-09-24 16:06:14 on dist-test-slave-n106"
server_key: "bc6d4bb855494482cd8819b9793d5192"
server_key_iv: "3d65695211eafbc9916998017b94ddbc"
server_key_version: "encryptionkey@0"
I20260924 16:06:14.254525 14637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "c8f5c6a78d2d4607b1f8ef0cb09c1429"
format_stamp: "Formatted at 2026-09-24 16:06:14 on dist-test-slave-n106"
server_key: "bc6d4bb855494482cd8819b9793d5192"
server_key_iv: "3d65695211eafbc9916998017b94ddbc"
server_key_version: "encryptionkey@0"
I20260924 16:06:14.258242 14637 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 16:06:14.260679 14652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:14.261869 14637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:06:14.262012 14637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "c8f5c6a78d2d4607b1f8ef0cb09c1429"
format_stamp: "Formatted at 2026-09-24 16:06:14 on dist-test-slave-n106"
server_key: "bc6d4bb855494482cd8819b9793d5192"
server_key_iv: "3d65695211eafbc9916998017b94ddbc"
server_key_version: "encryptionkey@0"
I20260924 16:06:14.262131 14637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:14.267014 14637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:14.270442 14637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:14.270593 14637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:14.277278 14637 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34843
I20260924 16:06:14.277391 14704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34843 every 8 connection(s)
I20260924 16:06:14.278250 14637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:14.279510 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 14637
I20260924 16:06:14.279672 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:06:14.279976 26036 external_mini_cluster.cc:1428] Setting key 964761927f636ea8e7a2339353177bb8
I20260924 16:06:14.281528 14705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:14.285151 14708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:14.281355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60157 (local address 127.0.0.1:34843)
0924 16:06:14.281776 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:14.281792 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:14.281827 (+ 35us) server_negotiation.cc:407] Connection header received
0924 16:06:14.282451 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:14.282467 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:14.282755 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:14.283034 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:14.283920 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:14.283925 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:14.283936 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:14.283947 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:14.283999 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:14.284017 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:14.284129 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:14.284234 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 16:06:14.284397 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":144,"threads_started":1}
I20260924 16:06:14.288049 26036 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:06:14.288051 14705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8f5c6a78d2d4607b1f8ef0cb09c1429: Bootstrap starting.
I20260924 16:06:14.290783 14705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8f5c6a78d2d4607b1f8ef0cb09c1429: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:14.291610 14705 log.cc:824] T 00000000000000000000000000000000 P c8f5c6a78d2d4607b1f8ef0cb09c1429: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:14.292418 14708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:14.290719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55291 (local address 127.0.0.1:34843)
0924 16:06:14.290851 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:14.290856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:14.291081 (+ 225us) server_negotiation.cc:407] Connection header received
0924 16:06:14.291178 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:14.291181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:14.291215 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:14.291286 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:14.291994 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:14.291998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:14.292000 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:14.292011 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:14.292041 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:14.292051 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:14.292118 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:14.292256 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 16:06:14.292286 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 16:06:14.293788 14705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8f5c6a78d2d4607b1f8ef0cb09c1429: No bootstrap required, opened a new log
I20260924 16:06:14.297171 14705 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8f5c6a78d2d4607b1f8ef0cb09c1429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f5c6a78d2d4607b1f8ef0cb09c1429" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34843 } }
I20260924 16:06:14.297372 14705 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8f5c6a78d2d4607b1f8ef0cb09c1429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:14.297406 14705 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8f5c6a78d2d4607b1f8ef0cb09c1429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f5c6a78d2d4607b1f8ef0cb09c1429, State: Initialized, Role: FOLLOWER
I20260924 16:06:14.297755 14705 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8f5c6a78d2d4607b1f8ef0cb09c1429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f5c6a78d2d4607b1f8ef0cb09c1429" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34843 } }
I20260924 16:06:14.297860 14705 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8f5c6a78d2d4607b1f8ef0cb09c1429 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:14.297902 14705 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8f5c6a78d2d4607b1f8ef0cb09c1429 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:14.297968 14705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8f5c6a78d2d4607b1f8ef0cb09c1429 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:14.298718 14705 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8f5c6a78d2d4607b1f8ef0cb09c1429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f5c6a78d2d4607b1f8ef0cb09c1429" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34843 } }
I20260924 16:06:14.299000 14705 leader_election.cc:302] T 00000000000000000000000000000000 P c8f5c6a78d2d4607b1f8ef0cb09c1429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8f5c6a78d2d4607b1f8ef0cb09c1429; no voters:
I20260924 16:06:14.299253 14705 leader_election.cc:288] T 00000000000000000000000000000000 P c8f5c6a78d2d4607b1f8ef0cb09c1429 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:06:14.299373 14715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8f5c6a78d2d4607b1f8ef0cb09c1429 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:14.299625 14715 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8f5c6a78d2d4607b1f8ef0cb09c1429 [term 1 LEADER]: Becoming Leader. State: Replica: c8f5c6a78d2d4607b1f8ef0cb09c1429, State: Running, Role: LEADER
I20260924 16:06:14.299958 14715 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8f5c6a78d2d4607b1f8ef0cb09c1429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f5c6a78d2d4607b1f8ef0cb09c1429" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34843 } }
I20260924 16:06:14.300364 14705 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8f5c6a78d2d4607b1f8ef0cb09c1429 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:14.301666 14717 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8f5c6a78d2d4607b1f8ef0cb09c1429 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8f5c6a78d2d4607b1f8ef0cb09c1429. Latest consensus state: current_term: 1 leader_uuid: "c8f5c6a78d2d4607b1f8ef0cb09c1429" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f5c6a78d2d4607b1f8ef0cb09c1429" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34843 } } }
I20260924 16:06:14.301664 14716 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8f5c6a78d2d4607b1f8ef0cb09c1429 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8f5c6a78d2d4607b1f8ef0cb09c1429" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f5c6a78d2d4607b1f8ef0cb09c1429" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34843 } } }
I20260924 16:06:14.301837 14717 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8f5c6a78d2d4607b1f8ef0cb09c1429 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:14.301854 14716 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8f5c6a78d2d4607b1f8ef0cb09c1429 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:14.302289 14727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:14.305598 14727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:14.311832 14727 catalog_manager.cc:1347] Generated new cluster ID: 648e8b9f4be64ea2bfcdd85ed7f30f02
I20260924 16:06:14.311908 14727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:14.322574 14727 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:14.322888 14727 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:14.339891 14727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8f5c6a78d2d4607b1f8ef0cb09c1429: Generated new TSK 0
I20260924 16:06:14.340639 14727 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:06:14.351850 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 14637
2026-09-24T16:06:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1405 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:06:14 dist-test-slave-n106 krb5kdc[14736](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:06:14 dist-test-slave-n106 krb5kdc[14736](info): set up 2 sockets
Sep 24 16:06:14 dist-test-slave-n106 krb5kdc[14736](info): commencing operation
krb5kdc: starting...
W20260924 16:06:15.389602 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:06:15 dist-test-slave-n106 krb5kdc[14736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:06:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:06:15.509661 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40837
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:15.619035 14749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:15.619287 14749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:15.619328 14749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:15.623337 14749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:15.623390 14749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:15.623430 14749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:15.623445 14749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:15.623461 14749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:15.628590 14749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:15.629521 14749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:15.630466 14749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:15.639091 14758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:15.639154 14749 server_base.cc:1034] running on GCE node
W20260924 16:06:15.639053 14755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:15.639081 14756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:15.639921 14749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:15.641052 14749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:15.642246 14749 hybrid_clock.cc:648] HybridClock initialized: now 1790265975642216 us; error 53 us; skew 500 ppm
Sep 24 16:06:15 dist-test-slave-n106 krb5kdc[14736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:15.653008 14749 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:06:15.654094 14749 webserver.cc:460] Webserver started at http://127.0.0.1:41629/ using document root <none> and password file <none>
I20260924 16:06:15.654722 14749 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:15.654824 14749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:15.655077 14749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:15.657256 14749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "29063aa69de84701aa7915796676ac86"
format_stamp: "Formatted at 2026-09-24 16:06:15 on dist-test-slave-n106"
server_key: "5f69ab2e05df4d73579d83d3dc98d660"
server_key_iv: "b6f1087fa1c9218907330b4ebccc0cfc"
server_key_version: "encryptionkey@0"
I20260924 16:06:15.657848 14749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "29063aa69de84701aa7915796676ac86"
format_stamp: "Formatted at 2026-09-24 16:06:15 on dist-test-slave-n106"
server_key: "5f69ab2e05df4d73579d83d3dc98d660"
server_key_iv: "b6f1087fa1c9218907330b4ebccc0cfc"
server_key_version: "encryptionkey@0"
I20260924 16:06:15.661695 14749 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:06:15.664090 14765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:15.665376 14749 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:06:15.665527 14749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "29063aa69de84701aa7915796676ac86"
format_stamp: "Formatted at 2026-09-24 16:06:15 on dist-test-slave-n106"
server_key: "5f69ab2e05df4d73579d83d3dc98d660"
server_key_iv: "b6f1087fa1c9218907330b4ebccc0cfc"
server_key_version: "encryptionkey@0"
I20260924 16:06:15.665656 14749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:15.689014 14749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:15.691934 14749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:15.692155 14749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:15.699337 14749 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40837
I20260924 16:06:15.699362 14817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40837 every 8 connection(s)
I20260924 16:06:15.700307 14749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:15.703027 14818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:15.706413 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 14749
I20260924 16:06:15.706568 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:06:15.706897 26036 external_mini_cluster.cc:1428] Setting key 754381042ff567597db7a9f9f6b2fc4a
I20260924 16:06:15.709127 14818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29063aa69de84701aa7915796676ac86: Bootstrap starting.
I20260924 16:06:15.711265 14818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29063aa69de84701aa7915796676ac86: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:15.711994 14818 log.cc:824] T 00000000000000000000000000000000 P 29063aa69de84701aa7915796676ac86: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:15.713868 14818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29063aa69de84701aa7915796676ac86: No bootstrap required, opened a new log
Sep 24 16:06:15 dist-test-slave-n106 krb5kdc[14736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 16:06:15.718078 14818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29063aa69de84701aa7915796676ac86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29063aa69de84701aa7915796676ac86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40837 } }
I20260924 16:06:15.718297 14818 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29063aa69de84701aa7915796676ac86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:15.718355 14818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29063aa69de84701aa7915796676ac86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29063aa69de84701aa7915796676ac86, State: Initialized, Role: FOLLOWER
I20260924 16:06:15.718858 14818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29063aa69de84701aa7915796676ac86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29063aa69de84701aa7915796676ac86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40837 } }
I20260924 16:06:15.719023 14818 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29063aa69de84701aa7915796676ac86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:15.719096 14818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29063aa69de84701aa7915796676ac86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:15.719228 14818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29063aa69de84701aa7915796676ac86 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:15.720310 14818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29063aa69de84701aa7915796676ac86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29063aa69de84701aa7915796676ac86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40837 } }
I20260924 16:06:15.720688 14818 leader_election.cc:302] T 00000000000000000000000000000000 P 29063aa69de84701aa7915796676ac86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29063aa69de84701aa7915796676ac86; no voters:
I20260924 16:06:15.721001 14818 leader_election.cc:288] T 00000000000000000000000000000000 P 29063aa69de84701aa7915796676ac86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:06:15.721122 14823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29063aa69de84701aa7915796676ac86 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:15.721333 14823 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29063aa69de84701aa7915796676ac86 [term 1 LEADER]: Becoming Leader. State: Replica: 29063aa69de84701aa7915796676ac86, State: Running, Role: LEADER
I20260924 16:06:15.721531 14821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:15.708360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:46095 (local address 127.0.0.1:40837)
0924 16:06:15.708810 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:15.708825 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:15.708861 (+ 36us) server_negotiation.cc:407] Connection header received
0924 16:06:15.709545 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:15.709565 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:15.709911 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:15.710259 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:15.711084 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:15.712160 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:15.713104 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:15.713574 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:15.713751 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:15.716547 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:15.716552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:15.716566 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:15.716594 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:15.718237 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:15.718804 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:15.718809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:15.718813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:15.718917 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:15.719301 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:15.719305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:15.719307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:15.719809 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:15.720030 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:15.720280 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 16:06:15.720534 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":137,"threads_started":1}
I20260924 16:06:15.721676 14823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29063aa69de84701aa7915796676ac86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29063aa69de84701aa7915796676ac86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40837 } }
I20260924 16:06:15.722254 14818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29063aa69de84701aa7915796676ac86 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:15.724195 14824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29063aa69de84701aa7915796676ac86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29063aa69de84701aa7915796676ac86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29063aa69de84701aa7915796676ac86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40837 } } }
I20260924 16:06:15.724318 14824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29063aa69de84701aa7915796676ac86 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:15.725365 26036 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:06:15.725418 14825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29063aa69de84701aa7915796676ac86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29063aa69de84701aa7915796676ac86. Latest consensus state: current_term: 1 leader_uuid: "29063aa69de84701aa7915796676ac86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29063aa69de84701aa7915796676ac86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40837 } } }
I20260924 16:06:15.725520 14825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29063aa69de84701aa7915796676ac86 [sys.catalog]: This master's current role is: LEADER
W20260924 16:06:15.727263 14837 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 29063aa69de84701aa7915796676ac86: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:06:15.727353 14837 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:06:15.727484 14838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:15.728452 14838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:15.734347 14838 catalog_manager.cc:1347] Generated new cluster ID: dd5f8502ea104acc9d97ee71bbfa228c
I20260924 16:06:15.734404 14838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:15.739017 14821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:15.727884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:32821 (local address 127.0.0.1:40837)
0924 16:06:15.728126 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:15.728131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:15.728467 (+ 336us) server_negotiation.cc:407] Connection header received
0924 16:06:15.728604 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:15.728607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:15.728661 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:15.728749 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:15.729550 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:15.730349 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:15.731437 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:15.731792 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:15.731904 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:15.733399 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:15.733403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:15.733405 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:15.733430 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:15.737208 (+ 3778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:15.737762 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:15.737765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:15.737767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:15.737843 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:15.738189 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:15.738191 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:15.738193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:15.738374 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:15.738501 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:15.738781 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 16:06:15.738871 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260924 16:06:15.763046 14838 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:15.764214 14838 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:15.773030 14838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29063aa69de84701aa7915796676ac86: Generated new TSK 0
I20260924 16:06:15.773873 14838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:06:15.784186 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 14749
W20260924 16:06:15.789739 14734 connection.cc:535] client connection to 127.0.0.1:40837 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40837): Connection reset by peer (error 104)
2026-09-24T16:06:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1431 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:06:15 dist-test-slave-n106 krb5kdc[14849](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:06:15 dist-test-slave-n106 krb5kdc[14849](info): set up 2 sockets
Sep 24 16:06:15 dist-test-slave-n106 krb5kdc[14849](info): commencing operation
krb5kdc: starting...
W20260924 16:06:16.826195 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:06:16 dist-test-slave-n106 krb5kdc[14849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:06:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:06:16.950747 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34111
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:17.061856 14862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:17.062186 14862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:17.062274 14862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:17.066344 14862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:17.066457 14862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:17.066497 14862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:17.066535 14862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:17.066568 14862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:17.072042 14862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:17.073222 14862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:17.074277 14862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:17.083073 14870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:17.083073 14867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:17.083151 14862 server_base.cc:1034] running on GCE node
W20260924 16:06:17.083066 14868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:17.083922 14862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:17.085175 14862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:17.086374 14862 hybrid_clock.cc:648] HybridClock initialized: now 1790265977086340 us; error 44 us; skew 500 ppm
Sep 24 16:06:17 dist-test-slave-n106 krb5kdc[14849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:17.089159 14862 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:06:17.090034 14862 webserver.cc:460] Webserver started at http://127.0.0.1:37045/ using document root <none> and password file <none>
I20260924 16:06:17.090566 14862 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:17.090624 14862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:17.090793 14862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:17.092937 14862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "e776d13a1148447ba62e2729c91c260c"
format_stamp: "Formatted at 2026-09-24 16:06:17 on dist-test-slave-n106"
server_key: "782966c9cec184398d681fb48dc47cc7"
server_key_iv: "c1d719b346d70d75bad37c1da6bb928e"
server_key_version: "encryptionkey@0"
I20260924 16:06:17.093466 14862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "e776d13a1148447ba62e2729c91c260c"
format_stamp: "Formatted at 2026-09-24 16:06:17 on dist-test-slave-n106"
server_key: "782966c9cec184398d681fb48dc47cc7"
server_key_iv: "c1d719b346d70d75bad37c1da6bb928e"
server_key_version: "encryptionkey@0"
I20260924 16:06:17.097235 14862 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:06:17.099730 14877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:17.101050 14862 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:06:17.101181 14862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "e776d13a1148447ba62e2729c91c260c"
format_stamp: "Formatted at 2026-09-24 16:06:17 on dist-test-slave-n106"
server_key: "782966c9cec184398d681fb48dc47cc7"
server_key_iv: "c1d719b346d70d75bad37c1da6bb928e"
server_key_version: "encryptionkey@0"
I20260924 16:06:17.101276 14862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:17.127010 14862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:17.130007 14862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:17.130184 14862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:17.137424 14862 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34111
I20260924 16:06:17.137451 14932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34111 every 8 connection(s)
I20260924 16:06:17.138579 14862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:17.141563 14933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:17.147176 14933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e776d13a1148447ba62e2729c91c260c: Bootstrap starting.
I20260924 16:06:17.147220 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 14862
I20260924 16:06:17.147372 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:06:17.147696 26036 external_mini_cluster.cc:1428] Setting key 52034ce3e4ebae13a742359ea7ee56ed
I20260924 16:06:17.149843 14933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e776d13a1148447ba62e2729c91c260c: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:17.150564 14933 log.cc:824] T 00000000000000000000000000000000 P e776d13a1148447ba62e2729c91c260c: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:17.152534 14933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e776d13a1148447ba62e2729c91c260c: No bootstrap required, opened a new log
I20260924 16:06:17.156286 14933 raft_consensus.cc:348] T 00000000000000000000000000000000 P e776d13a1148447ba62e2729c91c260c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e776d13a1148447ba62e2729c91c260c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34111 } }
I20260924 16:06:17.156603 14933 raft_consensus.cc:374] T 00000000000000000000000000000000 P e776d13a1148447ba62e2729c91c260c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:17.156687 14933 raft_consensus.cc:729] T 00000000000000000000000000000000 P e776d13a1148447ba62e2729c91c260c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e776d13a1148447ba62e2729c91c260c, State: Initialized, Role: FOLLOWER
I20260924 16:06:17.156908 14936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:17.149181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:33305 (local address 127.0.0.1:34111)
0924 16:06:17.149586 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:17.149611 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:17.149648 (+ 37us) server_negotiation.cc:407] Connection header received
0924 16:06:17.150339 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:17.150359 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:17.150749 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:17.151114 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:17.151519 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:17.152529 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:17.153438 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:17.153921 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:17.154113 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:17.155050 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:17.155055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:17.155068 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:17.155448 (+ 380us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:17.155501 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:17.155524 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:17.155668 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:17.155854 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 16:06:17.155992 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":129,"threads_started":1}
I20260924 16:06:17.157202 14933 consensus_queue.cc:260] T 00000000000000000000000000000000 P e776d13a1148447ba62e2729c91c260c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e776d13a1148447ba62e2729c91c260c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34111 } }
I20260924 16:06:17.157374 14933 raft_consensus.cc:388] T 00000000000000000000000000000000 P e776d13a1148447ba62e2729c91c260c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:17.157495 14933 raft_consensus.cc:482] T 00000000000000000000000000000000 P e776d13a1148447ba62e2729c91c260c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:17.157598 14933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e776d13a1148447ba62e2729c91c260c [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:17.158663 14933 raft_consensus.cc:504] T 00000000000000000000000000000000 P e776d13a1148447ba62e2729c91c260c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e776d13a1148447ba62e2729c91c260c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34111 } }
I20260924 16:06:17.159025 14933 leader_election.cc:302] T 00000000000000000000000000000000 P e776d13a1148447ba62e2729c91c260c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e776d13a1148447ba62e2729c91c260c; no voters:
I20260924 16:06:17.159456 14933 leader_election.cc:288] T 00000000000000000000000000000000 P e776d13a1148447ba62e2729c91c260c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:06:17.159541 14938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e776d13a1148447ba62e2729c91c260c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:17.159775 14938 raft_consensus.cc:686] T 00000000000000000000000000000000 P e776d13a1148447ba62e2729c91c260c [term 1 LEADER]: Becoming Leader. State: Replica: e776d13a1148447ba62e2729c91c260c, State: Running, Role: LEADER
I20260924 16:06:17.160161 26036 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:06:17.160130 14938 consensus_queue.cc:237] T 00000000000000000000000000000000 P e776d13a1148447ba62e2729c91c260c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e776d13a1148447ba62e2729c91c260c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34111 } }
I20260924 16:06:17.160743 14933 sys_catalog.cc:564] T 00000000000000000000000000000000 P e776d13a1148447ba62e2729c91c260c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:17.162714 14940 sys_catalog.cc:455] T 00000000000000000000000000000000 P e776d13a1148447ba62e2729c91c260c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e776d13a1148447ba62e2729c91c260c. Latest consensus state: current_term: 1 leader_uuid: "e776d13a1148447ba62e2729c91c260c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e776d13a1148447ba62e2729c91c260c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34111 } } }
I20260924 16:06:17.162846 14940 sys_catalog.cc:458] T 00000000000000000000000000000000 P e776d13a1148447ba62e2729c91c260c [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:17.163190 14939 sys_catalog.cc:455] T 00000000000000000000000000000000 P e776d13a1148447ba62e2729c91c260c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e776d13a1148447ba62e2729c91c260c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e776d13a1148447ba62e2729c91c260c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34111 } } }
I20260924 16:06:17.163506 14939 sys_catalog.cc:458] T 00000000000000000000000000000000 P e776d13a1148447ba62e2729c91c260c [sys.catalog]: This master's current role is: LEADER
W20260924 16:06:17.166220 14957 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e776d13a1148447ba62e2729c91c260c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:06:17.166298 14957 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:06:17.166375 14958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:17.167341 14958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:17.168123 14936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:17.162818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:58461 (local address 127.0.0.1:34111)
0924 16:06:17.162977 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:17.162983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:17.163520 (+ 537us) server_negotiation.cc:407] Connection header received
0924 16:06:17.163734 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:17.163737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:17.163787 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:17.163872 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:17.164141 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:17.164895 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:17.166046 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:17.166402 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:17.166501 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:17.167597 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:17.167601 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:17.167604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:17.167618 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:17.167647 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:17.167660 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:17.167751 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:17.167942 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 16:06:17.167980 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 16:06:17.173039 14958 catalog_manager.cc:1347] Generated new cluster ID: e5a98eb2ca174b9cbc5c22bc3eece6a5
I20260924 16:06:17.173144 14958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:17.223130 14958 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:17.224510 14958 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:17.234462 14958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e776d13a1148447ba62e2729c91c260c: Generated new TSK 0
I20260924 16:06:17.235328 14958 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:06:17.239708 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 14862
W20260924 16:06:17.245903 14847 connection.cc:535] client connection to 127.0.0.1:34111 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34111): Connection reset by peer (error 104)
2026-09-24T16:06:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1456 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:06:17 dist-test-slave-n106 krb5kdc[14964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:06:17 dist-test-slave-n106 krb5kdc[14964](info): set up 2 sockets
Sep 24 16:06:17 dist-test-slave-n106 krb5kdc[14964](info): commencing operation
krb5kdc: starting...
W20260924 16:06:18.277632 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:06:18 dist-test-slave-n106 krb5kdc[14964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:06:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:06:18.403482 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35983
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:18.518312 14977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:18.518581 14977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:18.518623 14977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:18.522660 14977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:18.522723 14977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:18.522742 14977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:18.522759 14977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:18.522804 14977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:18.528017 14977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:18.529119 14977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:18.530282 14977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:18.538733 14982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:18.538801 14983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:18.538786 14977 server_base.cc:1034] running on GCE node
W20260924 16:06:18.538741 14985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:18.539577 14977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:18.540666 14977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:18.541829 14977 hybrid_clock.cc:648] HybridClock initialized: now 1790265978541799 us; error 40 us; skew 500 ppm
Sep 24 16:06:18 dist-test-slave-n106 krb5kdc[14964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:18.544556 14977 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:06:18.545459 14977 webserver.cc:460] Webserver started at http://127.0.0.1:38419/ using document root <none> and password file <none>
I20260924 16:06:18.546011 14977 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:18.546115 14977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:18.546324 14977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:18.548303 14977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "e928fc77b2ad4d29a7736c51cf8e22ac"
format_stamp: "Formatted at 2026-09-24 16:06:18 on dist-test-slave-n106"
server_key: "d17c03cfd39c6424e2c3d65dced52cf7"
server_key_iv: "746af4339572b9be48d9a3f193ace7d4"
server_key_version: "encryptionkey@0"
I20260924 16:06:18.548923 14977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "e928fc77b2ad4d29a7736c51cf8e22ac"
format_stamp: "Formatted at 2026-09-24 16:06:18 on dist-test-slave-n106"
server_key: "d17c03cfd39c6424e2c3d65dced52cf7"
server_key_iv: "746af4339572b9be48d9a3f193ace7d4"
server_key_version: "encryptionkey@0"
I20260924 16:06:18.552548 14977 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:06:18.554876 14992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:18.556051 14977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:06:18.556191 14977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "e928fc77b2ad4d29a7736c51cf8e22ac"
format_stamp: "Formatted at 2026-09-24 16:06:18 on dist-test-slave-n106"
server_key: "d17c03cfd39c6424e2c3d65dced52cf7"
server_key_iv: "746af4339572b9be48d9a3f193ace7d4"
server_key_version: "encryptionkey@0"
I20260924 16:06:18.556304 14977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:18.560900 14977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:18.563694 14977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:18.563889 14977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:18.570958 14977 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35983
I20260924 16:06:18.570984 15045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35983 every 8 connection(s)
I20260924 16:06:18.571862 14977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:18.574513 15046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:18.579782 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 14977
I20260924 16:06:18.579923 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:06:18.580049 15046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e928fc77b2ad4d29a7736c51cf8e22ac: Bootstrap starting.
I20260924 16:06:18.580274 26036 external_mini_cluster.cc:1428] Setting key fb5629e5f9b64e0ec8e9fc77e4ff06dd
I20260924 16:06:18.582644 15046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e928fc77b2ad4d29a7736c51cf8e22ac: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:18.583371 15046 log.cc:824] T 00000000000000000000000000000000 P e928fc77b2ad4d29a7736c51cf8e22ac: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:18.585548 15046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e928fc77b2ad4d29a7736c51cf8e22ac: No bootstrap required, opened a new log
I20260924 16:06:18.585596 15049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:18.581685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:37091 (local address 127.0.0.1:35983)
0924 16:06:18.582122 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:18.582137 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:18.582172 (+ 35us) server_negotiation.cc:407] Connection header received
0924 16:06:18.582881 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:18.582901 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:18.583190 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:18.583535 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:18.584333 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:18.584338 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:18.584350 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:18.584360 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:18.584451 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:18.584485 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:18.584577 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:18.584720 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 16:06:18.584893 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":129,"threads_started":1}
I20260924 16:06:18.588341 26036 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:06:18.589581 15046 raft_consensus.cc:348] T 00000000000000000000000000000000 P e928fc77b2ad4d29a7736c51cf8e22ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e928fc77b2ad4d29a7736c51cf8e22ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35983 } }
I20260924 16:06:18.589843 15046 raft_consensus.cc:374] T 00000000000000000000000000000000 P e928fc77b2ad4d29a7736c51cf8e22ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:18.589920 15046 raft_consensus.cc:729] T 00000000000000000000000000000000 P e928fc77b2ad4d29a7736c51cf8e22ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e928fc77b2ad4d29a7736c51cf8e22ac, State: Initialized, Role: FOLLOWER
I20260924 16:06:18.590384 15046 consensus_queue.cc:260] T 00000000000000000000000000000000 P e928fc77b2ad4d29a7736c51cf8e22ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e928fc77b2ad4d29a7736c51cf8e22ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35983 } }
I20260924 16:06:18.590612 15046 raft_consensus.cc:388] T 00000000000000000000000000000000 P e928fc77b2ad4d29a7736c51cf8e22ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:18.590703 15046 raft_consensus.cc:482] T 00000000000000000000000000000000 P e928fc77b2ad4d29a7736c51cf8e22ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:18.590817 15046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e928fc77b2ad4d29a7736c51cf8e22ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:18.591997 15049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:18.590414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:58787 (local address 127.0.0.1:35983)
0924 16:06:18.590535 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:18.590547 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:18.590764 (+ 217us) server_negotiation.cc:407] Connection header received
0924 16:06:18.590876 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:18.590879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:18.590933 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:18.591010 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:18.591661 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:18.591664 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:18.591666 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:18.591676 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:18.591700 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:18.591710 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:18.591776 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:18.591887 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 16:06:18.591917 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 16:06:18.592082 15046 raft_consensus.cc:504] T 00000000000000000000000000000000 P e928fc77b2ad4d29a7736c51cf8e22ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e928fc77b2ad4d29a7736c51cf8e22ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35983 } }
I20260924 16:06:18.592476 15046 leader_election.cc:302] T 00000000000000000000000000000000 P e928fc77b2ad4d29a7736c51cf8e22ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e928fc77b2ad4d29a7736c51cf8e22ac; no voters:
I20260924 16:06:18.592820 15046 leader_election.cc:288] T 00000000000000000000000000000000 P e928fc77b2ad4d29a7736c51cf8e22ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:06:18.592900 15056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e928fc77b2ad4d29a7736c51cf8e22ac [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:18.593125 15056 raft_consensus.cc:686] T 00000000000000000000000000000000 P e928fc77b2ad4d29a7736c51cf8e22ac [term 1 LEADER]: Becoming Leader. State: Replica: e928fc77b2ad4d29a7736c51cf8e22ac, State: Running, Role: LEADER
I20260924 16:06:18.593448 15056 consensus_queue.cc:237] T 00000000000000000000000000000000 P e928fc77b2ad4d29a7736c51cf8e22ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e928fc77b2ad4d29a7736c51cf8e22ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35983 } }
I20260924 16:06:18.594067 15046 sys_catalog.cc:564] T 00000000000000000000000000000000 P e928fc77b2ad4d29a7736c51cf8e22ac [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:18.595085 15057 sys_catalog.cc:455] T 00000000000000000000000000000000 P e928fc77b2ad4d29a7736c51cf8e22ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader e928fc77b2ad4d29a7736c51cf8e22ac. Latest consensus state: current_term: 1 leader_uuid: "e928fc77b2ad4d29a7736c51cf8e22ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e928fc77b2ad4d29a7736c51cf8e22ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35983 } } }
I20260924 16:06:18.595211 15057 sys_catalog.cc:458] T 00000000000000000000000000000000 P e928fc77b2ad4d29a7736c51cf8e22ac [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:18.595319 15058 sys_catalog.cc:455] T 00000000000000000000000000000000 P e928fc77b2ad4d29a7736c51cf8e22ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e928fc77b2ad4d29a7736c51cf8e22ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e928fc77b2ad4d29a7736c51cf8e22ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35983 } } }
I20260924 16:06:18.595602 15058 sys_catalog.cc:458] T 00000000000000000000000000000000 P e928fc77b2ad4d29a7736c51cf8e22ac [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:18.596360 15065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:18.598958 15065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:18.603777 15065 catalog_manager.cc:1347] Generated new cluster ID: d7dafd80207d4133b75e6a9024370cff
I20260924 16:06:18.603838 15065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:18.616763 15065 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:18.617107 15065 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:18.632900 15065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e928fc77b2ad4d29a7736c51cf8e22ac: Generated new TSK 0
I20260924 16:06:18.633570 15065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:06:18.637697 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 14977
2026-09-24T16:06:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1397 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:06:18 dist-test-slave-n106 krb5kdc[15078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:06:18 dist-test-slave-n106 krb5kdc[15078](info): set up 2 sockets
Sep 24 16:06:18 dist-test-slave-n106 krb5kdc[15078](info): commencing operation
krb5kdc: starting...
W20260924 16:06:19.681630 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:06:19 dist-test-slave-n106 krb5kdc[15078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:06:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:06:19.802696 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39591
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:19.913710 15091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:19.914063 15091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:19.914247 15091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:19.918535 15091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:19.918642 15091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:19.918684 15091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:19.918720 15091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:19.918756 15091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:19.924239 15091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39591
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:19.925455 15091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:19.926571 15091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:19.935603 15099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:19.935617 15096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:19.935849 15091 server_base.cc:1034] running on GCE node
W20260924 16:06:19.935616 15097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:19.936609 15091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:19.937801 15091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:19.939037 15091 hybrid_clock.cc:648] HybridClock initialized: now 1790265979939020 us; error 25 us; skew 500 ppm
Sep 24 16:06:19 dist-test-slave-n106 krb5kdc[15078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:19.941871 15091 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:06:19.942786 15091 webserver.cc:460] Webserver started at http://127.0.0.1:43397/ using document root <none> and password file <none>
I20260924 16:06:19.943382 15091 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:19.943492 15091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:19.943732 15091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:19.945861 15091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "f8f99c247e0244c093698efd13e4c891"
format_stamp: "Formatted at 2026-09-24 16:06:19 on dist-test-slave-n106"
server_key: "ffea79b5be5fe9e78644722065dc37b5"
server_key_iv: "f9f3eae734a631771ccb4477c8c5cff4"
server_key_version: "encryptionkey@0"
I20260924 16:06:19.946440 15091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "f8f99c247e0244c093698efd13e4c891"
format_stamp: "Formatted at 2026-09-24 16:06:19 on dist-test-slave-n106"
server_key: "ffea79b5be5fe9e78644722065dc37b5"
server_key_iv: "f9f3eae734a631771ccb4477c8c5cff4"
server_key_version: "encryptionkey@0"
I20260924 16:06:19.950177 15091 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260924 16:06:19.952702 15106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:19.953912 15091 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:06:19.954057 15091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "f8f99c247e0244c093698efd13e4c891"
format_stamp: "Formatted at 2026-09-24 16:06:19 on dist-test-slave-n106"
server_key: "ffea79b5be5fe9e78644722065dc37b5"
server_key_iv: "f9f3eae734a631771ccb4477c8c5cff4"
server_key_version: "encryptionkey@0"
I20260924 16:06:19.954185 15091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:19.967845 15091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:19.970700 15091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:19.970906 15091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:19.977794 15091 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39591
I20260924 16:06:19.977808 15160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39591 every 8 connection(s)
I20260924 16:06:19.978750 15091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:19.979106 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 15091
I20260924 16:06:19.979269 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:06:19.979583 26036 external_mini_cluster.cc:1428] Setting key d5c0539f9475c3cdac6e580a4ff61d9f
I20260924 16:06:19.981945 15161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 16:06:19 dist-test-slave-n106 krb5kdc[15078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 16:06:19.988739 15161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8f99c247e0244c093698efd13e4c891: Bootstrap starting.
I20260924 16:06:19.991525 15161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8f99c247e0244c093698efd13e4c891: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:19.992262 15161 log.cc:824] T 00000000000000000000000000000000 P f8f99c247e0244c093698efd13e4c891: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:19.994100 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:19.980971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:37375 (local address 127.0.0.1:39591)
0924 16:06:19.981417 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:19.981432 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:19.981472 (+ 40us) server_negotiation.cc:407] Connection header received
0924 16:06:19.982147 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:19.982166 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:19.982495 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:19.982768 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:19.983683 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:19.984731 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:19.985902 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:19.986347 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:19.986463 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:19.989313 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:19.989320 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:19.989335 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:19.989365 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:19.991216 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:19.991724 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:19.991729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:19.991734 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:19.991842 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:19.992187 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:19.992191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:19.992193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:19.992655 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:19.992864 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:19.993170 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 16:06:19.993429 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":148,"threads_started":1}
I20260924 16:06:19.994340 15161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8f99c247e0244c093698efd13e4c891: No bootstrap required, opened a new log
I20260924 16:06:19.997145 26036 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:06:19.998811 15161 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8f99c247e0244c093698efd13e4c891 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8f99c247e0244c093698efd13e4c891" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39591 } }
I20260924 16:06:19.999094 15161 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8f99c247e0244c093698efd13e4c891 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:19.999161 15161 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8f99c247e0244c093698efd13e4c891 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8f99c247e0244c093698efd13e4c891, State: Initialized, Role: FOLLOWER
I20260924 16:06:19.999634 15161 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8f99c247e0244c093698efd13e4c891 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8f99c247e0244c093698efd13e4c891" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39591 } }
I20260924 16:06:19.999811 15161 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8f99c247e0244c093698efd13e4c891 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:19.999878 15161 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8f99c247e0244c093698efd13e4c891 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:19.999981 15161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8f99c247e0244c093698efd13e4c891 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:20.001082 15161 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8f99c247e0244c093698efd13e4c891 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8f99c247e0244c093698efd13e4c891" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39591 } }
I20260924 16:06:20.001432 15161 leader_election.cc:302] T 00000000000000000000000000000000 P f8f99c247e0244c093698efd13e4c891 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8f99c247e0244c093698efd13e4c891; no voters:
I20260924 16:06:20.001778 15161 leader_election.cc:288] T 00000000000000000000000000000000 P f8f99c247e0244c093698efd13e4c891 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:06:20.001880 15171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8f99c247e0244c093698efd13e4c891 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:20.002120 15171 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8f99c247e0244c093698efd13e4c891 [term 1 LEADER]: Becoming Leader. State: Replica: f8f99c247e0244c093698efd13e4c891, State: Running, Role: LEADER
I20260924 16:06:20.002460 15171 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8f99c247e0244c093698efd13e4c891 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8f99c247e0244c093698efd13e4c891" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39591 } }
I20260924 16:06:20.003027 15161 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8f99c247e0244c093698efd13e4c891 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:20.004305 15172 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8f99c247e0244c093698efd13e4c891 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8f99c247e0244c093698efd13e4c891" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8f99c247e0244c093698efd13e4c891" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39591 } } }
I20260924 16:06:20.004464 15172 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8f99c247e0244c093698efd13e4c891 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:20.004794 15173 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8f99c247e0244c093698efd13e4c891 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8f99c247e0244c093698efd13e4c891. Latest consensus state: current_term: 1 leader_uuid: "f8f99c247e0244c093698efd13e4c891" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8f99c247e0244c093698efd13e4c891" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39591 } } }
I20260924 16:06:20.004877 15180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:20.004966 15173 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8f99c247e0244c093698efd13e4c891 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:20.008191 15180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:20.008701 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:19.999403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:39243 (local address 127.0.0.1:39591)
0924 16:06:19.999512 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:19.999516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:19.999730 (+ 214us) server_negotiation.cc:407] Connection header received
0924 16:06:19.999878 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:19.999881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:19.999938 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:20.000019 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:20.000723 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.001508 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.002696 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.003015 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.003118 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:20.004938 (+ 1820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:20.004942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:20.004945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:20.004976 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:20.006735 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:20.007286 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:20.007289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:20.007291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:20.007381 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:20.007758 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:20.007761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:20.007763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:20.007972 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:20.008120 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:20.008447 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 16:06:20.008547 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 16:06:20.016181 15180 catalog_manager.cc:1347] Generated new cluster ID: 8fc7f49c858c47fbb431cbf1da716fb9
I20260924 16:06:20.016245 15180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:20.031850 15180 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:20.032914 15180 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:20.043236 15180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8f99c247e0244c093698efd13e4c891: Generated new TSK 0
I20260924 16:06:20.043946 15180 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:06:20.053052 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 15091
W20260924 16:06:20.059190 15076 connection.cc:535] client connection to 127.0.0.1:39591 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39591): Connection reset by peer (error 104)
2026-09-24T16:06:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:06:20 dist-test-slave-n106 krb5kdc[15192](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:06:20 dist-test-slave-n106 krb5kdc[15192](info): set up 2 sockets
Sep 24 16:06:20 dist-test-slave-n106 krb5kdc[15192](info): commencing operation
krb5kdc: starting...
W20260924 16:06:21.097539 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:06:21 dist-test-slave-n106 krb5kdc[15192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:06:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:06:21.215667 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46767
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:21.325755 15205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:21.326031 15205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:21.326115 15205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:21.330071 15205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:21.330143 15205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:21.330181 15205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:21.330237 15205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:21.330276 15205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:21.335290 15205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46767
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:21.336360 15205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:21.337365 15205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:21.345731 15212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:21.345793 15211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:21.345722 15214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:21.345981 15205 server_base.cc:1034] running on GCE node
I20260924 16:06:21.346568 15205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:21.347646 15205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:21.348834 15205 hybrid_clock.cc:648] HybridClock initialized: now 1790265981348815 us; error 30 us; skew 500 ppm
Sep 24 16:06:21 dist-test-slave-n106 krb5kdc[15192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:21.357004 15205 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:06:21.357920 15205 webserver.cc:460] Webserver started at http://127.0.0.1:45839/ using document root <none> and password file <none>
I20260924 16:06:21.358443 15205 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:21.358500 15205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:21.358671 15205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:21.360735 15205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "c3721c63ec514592b4f7b2a97f1bd03f"
format_stamp: "Formatted at 2026-09-24 16:06:21 on dist-test-slave-n106"
server_key: "28a7b381285384385cd51cdd904cd16c"
server_key_iv: "c0149d850036ef718cd5df4a7c6a665b"
server_key_version: "encryptionkey@0"
I20260924 16:06:21.361231 15205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "c3721c63ec514592b4f7b2a97f1bd03f"
format_stamp: "Formatted at 2026-09-24 16:06:21 on dist-test-slave-n106"
server_key: "28a7b381285384385cd51cdd904cd16c"
server_key_iv: "c0149d850036ef718cd5df4a7c6a665b"
server_key_version: "encryptionkey@0"
I20260924 16:06:21.364856 15205 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260924 16:06:21.367173 15221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:21.368383 15205 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:06:21.368573 15205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "c3721c63ec514592b4f7b2a97f1bd03f"
format_stamp: "Formatted at 2026-09-24 16:06:21 on dist-test-slave-n106"
server_key: "28a7b381285384385cd51cdd904cd16c"
server_key_iv: "c0149d850036ef718cd5df4a7c6a665b"
server_key_version: "encryptionkey@0"
I20260924 16:06:21.368686 15205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:21.396891 15205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:21.400364 15205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:21.400606 15205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:21.407805 15205 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46767
I20260924 16:06:21.407829 15274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46767 every 8 connection(s)
I20260924 16:06:21.408842 15205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:21.411823 15275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:21.411823 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 15205
I20260924 16:06:21.411970 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:06:21.412317 26036 external_mini_cluster.cc:1428] Setting key 028d99ab0279ae1276ff36f7ba66fb46
I20260924 16:06:21.418483 15275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3721c63ec514592b4f7b2a97f1bd03f: Bootstrap starting.
I20260924 16:06:21.420892 15275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3721c63ec514592b4f7b2a97f1bd03f: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:21.421558 15278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:21.413834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:49679 (local address 127.0.0.1:46767)
0924 16:06:21.414285 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:21.414300 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:21.414337 (+ 37us) server_negotiation.cc:407] Connection header received
0924 16:06:21.415217 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:21.415236 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:21.415577 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:21.415914 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:21.416312 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:21.417333 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:21.418223 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:21.418733 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:21.418893 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:21.419799 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:21.419804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:21.419816 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:21.420251 (+ 435us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:21.420322 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:21.420346 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:21.420526 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:21.420666 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 16:06:21.420798 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":145,"threads_started":1}
I20260924 16:06:21.421687 15275 log.cc:824] T 00000000000000000000000000000000 P c3721c63ec514592b4f7b2a97f1bd03f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:21.423746 15275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3721c63ec514592b4f7b2a97f1bd03f: No bootstrap required, opened a new log
I20260924 16:06:21.424469 26036 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:06:21.427866 15275 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3721c63ec514592b4f7b2a97f1bd03f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3721c63ec514592b4f7b2a97f1bd03f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46767 } }
I20260924 16:06:21.428153 15275 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3721c63ec514592b4f7b2a97f1bd03f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:21.428244 15275 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3721c63ec514592b4f7b2a97f1bd03f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3721c63ec514592b4f7b2a97f1bd03f, State: Initialized, Role: FOLLOWER
I20260924 16:06:21.428714 15275 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3721c63ec514592b4f7b2a97f1bd03f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3721c63ec514592b4f7b2a97f1bd03f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46767 } }
I20260924 16:06:21.428836 15275 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3721c63ec514592b4f7b2a97f1bd03f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:21.428927 15275 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3721c63ec514592b4f7b2a97f1bd03f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:21.429025 15275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3721c63ec514592b4f7b2a97f1bd03f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:21.430095 15275 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3721c63ec514592b4f7b2a97f1bd03f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3721c63ec514592b4f7b2a97f1bd03f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46767 } }
I20260924 16:06:21.430481 15275 leader_election.cc:302] T 00000000000000000000000000000000 P c3721c63ec514592b4f7b2a97f1bd03f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3721c63ec514592b4f7b2a97f1bd03f; no voters:
I20260924 16:06:21.430686 15278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:21.426464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:45159 (local address 127.0.0.1:46767)
0924 16:06:21.426607 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:21.426611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:21.426701 (+ 90us) server_negotiation.cc:407] Connection header received
0924 16:06:21.426789 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:21.426793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:21.426839 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:21.426931 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:21.427131 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:21.427899 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:21.428908 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:21.429279 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:21.429410 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:21.430250 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:21.430254 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:21.430256 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:21.430269 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:21.430297 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:21.430310 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:21.430402 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:21.430513 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 16:06:21.430545 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 16:06:21.430864 15275 leader_election.cc:288] T 00000000000000000000000000000000 P c3721c63ec514592b4f7b2a97f1bd03f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:06:21.430954 15285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3721c63ec514592b4f7b2a97f1bd03f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:21.431178 15285 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3721c63ec514592b4f7b2a97f1bd03f [term 1 LEADER]: Becoming Leader. State: Replica: c3721c63ec514592b4f7b2a97f1bd03f, State: Running, Role: LEADER
I20260924 16:06:21.431468 15285 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3721c63ec514592b4f7b2a97f1bd03f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3721c63ec514592b4f7b2a97f1bd03f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46767 } }
I20260924 16:06:21.432154 15275 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3721c63ec514592b4f7b2a97f1bd03f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:21.432780 15287 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3721c63ec514592b4f7b2a97f1bd03f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3721c63ec514592b4f7b2a97f1bd03f. Latest consensus state: current_term: 1 leader_uuid: "c3721c63ec514592b4f7b2a97f1bd03f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3721c63ec514592b4f7b2a97f1bd03f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46767 } } }
I20260924 16:06:21.432917 15287 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3721c63ec514592b4f7b2a97f1bd03f [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:21.433281 15286 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3721c63ec514592b4f7b2a97f1bd03f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3721c63ec514592b4f7b2a97f1bd03f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3721c63ec514592b4f7b2a97f1bd03f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46767 } } }
I20260924 16:06:21.433383 15292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:21.434075 15286 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3721c63ec514592b4f7b2a97f1bd03f [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:21.436830 15292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:21.442314 15292 catalog_manager.cc:1347] Generated new cluster ID: 9b124b1757104d08b912c099d2479cbe
I20260924 16:06:21.442376 15292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:21.458848 15292 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:21.460101 15292 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:21.464810 15292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3721c63ec514592b4f7b2a97f1bd03f: Generated new TSK 0
I20260924 16:06:21.465538 15292 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:06:21.474517 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 15205
W20260924 16:06:21.480392 15190 connection.cc:535] client connection to 127.0.0.1:46767 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46767): Connection reset by peer (error 104)
2026-09-24T16:06:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1421 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:06:21 dist-test-slave-n106 krb5kdc[15306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:06:21 dist-test-slave-n106 krb5kdc[15306](info): set up 2 sockets
Sep 24 16:06:21 dist-test-slave-n106 krb5kdc[15306](info): commencing operation
krb5kdc: starting...
W20260924 16:06:22.521574 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:06:22 dist-test-slave-n106 krb5kdc[15306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:06:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:06:22.639712 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39135
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:22.750720 15319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:22.751034 15319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:22.751137 15319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:22.755226 15319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:22.755321 15319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:22.755362 15319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:22.755421 15319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:22.755463 15319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:22.760849 15319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39135
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265708917108-26036-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:22.761938 15319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:22.762991 15319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:06:22.771651 15319 server_base.cc:1034] running on GCE node
W20260924 16:06:22.771727 15327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:22.771566 15325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:22.771566 15324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:22.772372 15319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:22.773521 15319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:22.774708 15319 hybrid_clock.cc:648] HybridClock initialized: now 1790265982774673 us; error 45 us; skew 500 ppm
Sep 24 16:06:22 dist-test-slave-n106 krb5kdc[15306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:22.777503 15319 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:06:22.778328 15319 webserver.cc:460] Webserver started at http://127.0.0.1:35709/ using document root <none> and password file <none>
I20260924 16:06:22.778888 15319 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:22.778978 15319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:22.779212 15319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:22.781284 15319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "6a284cc77cc1461abfb3237b3fe3ea1b"
format_stamp: "Formatted at 2026-09-24 16:06:22 on dist-test-slave-n106"
server_key: "6530908a94de23438c272c2784101f81"
server_key_iv: "9cca9b9e80758ae391d5d3727f61d6b4"
server_key_version: "encryptionkey@0"
I20260924 16:06:22.781842 15319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "6a284cc77cc1461abfb3237b3fe3ea1b"
format_stamp: "Formatted at 2026-09-24 16:06:22 on dist-test-slave-n106"
server_key: "6530908a94de23438c272c2784101f81"
server_key_iv: "9cca9b9e80758ae391d5d3727f61d6b4"
server_key_version: "encryptionkey@0"
I20260924 16:06:22.785491 15319 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:06:22.787881 15334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:22.789098 15319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:06:22.789237 15319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "6a284cc77cc1461abfb3237b3fe3ea1b"
format_stamp: "Formatted at 2026-09-24 16:06:22 on dist-test-slave-n106"
server_key: "6530908a94de23438c272c2784101f81"
server_key_iv: "9cca9b9e80758ae391d5d3727f61d6b4"
server_key_version: "encryptionkey@0"
I20260924 16:06:22.789347 15319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:22.793803 15319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:22.796916 15319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:22.797106 15319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:22.803442 15319 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39135
I20260924 16:06:22.803509 15388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39135 every 8 connection(s)
I20260924 16:06:22.804399 15319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:22.805860 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 15319
I20260924 16:06:22.805998 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:06:22.806329 26036 external_mini_cluster.cc:1428] Setting key 4f1abaa0bef40969a60d060dae3a35ab
I20260924 16:06:22.807546 15389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:22.811422 15392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:22.807807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:41139 (local address 127.0.0.1:39135)
0924 16:06:22.808204 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:22.808220 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:22.808253 (+ 33us) server_negotiation.cc:407] Connection header received
0924 16:06:22.808929 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:22.808945 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:22.809214 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:22.809540 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:22.810300 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:22.810305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:22.810317 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:22.810327 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:22.810378 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:22.810397 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:22.810545 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:22.810625 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 16:06:22.810786 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":132,"threads_started":1}
I20260924 16:06:22.813522 15389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a284cc77cc1461abfb3237b3fe3ea1b: Bootstrap starting.
I20260924 16:06:22.814137 26036 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:06:22.815850 15389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a284cc77cc1461abfb3237b3fe3ea1b: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:22.816797 15389 log.cc:824] T 00000000000000000000000000000000 P 6a284cc77cc1461abfb3237b3fe3ea1b: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:22.817886 15392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:22.816466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:38249 (local address 127.0.0.1:39135)
0924 16:06:22.816591 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:22.816596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:22.816610 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:06:22.816648 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:22.816654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:22.816692 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:22.816774 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:22.817463 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:22.817467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:22.817470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:22.817484 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:22.817514 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:22.817530 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:22.817628 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:22.817722 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 16:06:22.817753 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 16:06:22.818887 15389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a284cc77cc1461abfb3237b3fe3ea1b: No bootstrap required, opened a new log
I20260924 16:06:22.822275 15389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a284cc77cc1461abfb3237b3fe3ea1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a284cc77cc1461abfb3237b3fe3ea1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39135 } }
I20260924 16:06:22.822516 15389 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a284cc77cc1461abfb3237b3fe3ea1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:22.822602 15389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a284cc77cc1461abfb3237b3fe3ea1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a284cc77cc1461abfb3237b3fe3ea1b, State: Initialized, Role: FOLLOWER
I20260924 16:06:22.823055 15389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a284cc77cc1461abfb3237b3fe3ea1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a284cc77cc1461abfb3237b3fe3ea1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39135 } }
I20260924 16:06:22.823220 15389 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a284cc77cc1461abfb3237b3fe3ea1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:22.823307 15389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a284cc77cc1461abfb3237b3fe3ea1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:22.823423 15389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a284cc77cc1461abfb3237b3fe3ea1b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:22.824259 15389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a284cc77cc1461abfb3237b3fe3ea1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a284cc77cc1461abfb3237b3fe3ea1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39135 } }
I20260924 16:06:22.824623 15389 leader_election.cc:302] T 00000000000000000000000000000000 P 6a284cc77cc1461abfb3237b3fe3ea1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a284cc77cc1461abfb3237b3fe3ea1b; no voters:
I20260924 16:06:22.824922 15389 leader_election.cc:288] T 00000000000000000000000000000000 P 6a284cc77cc1461abfb3237b3fe3ea1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:06:22.825028 15399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a284cc77cc1461abfb3237b3fe3ea1b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:22.825321 15399 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a284cc77cc1461abfb3237b3fe3ea1b [term 1 LEADER]: Becoming Leader. State: Replica: 6a284cc77cc1461abfb3237b3fe3ea1b, State: Running, Role: LEADER
I20260924 16:06:22.825667 15399 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a284cc77cc1461abfb3237b3fe3ea1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a284cc77cc1461abfb3237b3fe3ea1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39135 } }
I20260924 16:06:22.826141 15389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a284cc77cc1461abfb3237b3fe3ea1b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:22.827414 15401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a284cc77cc1461abfb3237b3fe3ea1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a284cc77cc1461abfb3237b3fe3ea1b. Latest consensus state: current_term: 1 leader_uuid: "6a284cc77cc1461abfb3237b3fe3ea1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a284cc77cc1461abfb3237b3fe3ea1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39135 } } }
I20260924 16:06:22.827535 15401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a284cc77cc1461abfb3237b3fe3ea1b [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:22.827502 15400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a284cc77cc1461abfb3237b3fe3ea1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a284cc77cc1461abfb3237b3fe3ea1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a284cc77cc1461abfb3237b3fe3ea1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39135 } } }
I20260924 16:06:22.827600 15400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a284cc77cc1461abfb3237b3fe3ea1b [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:22.828034 15408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:22.830705 15408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:22.835930 15408 catalog_manager.cc:1347] Generated new cluster ID: d27b716d748348a3a8f0cfbc100f0087
I20260924 16:06:22.835990 15408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:22.850577 15408 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:22.850878 15408 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:22.856423 15408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a284cc77cc1461abfb3237b3fe3ea1b: Generated new TSK 0
I20260924 16:06:22.857005 15408 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:06:22.862659 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 15319
2026-09-24T16:06:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1387 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:06:22 dist-test-slave-n106 krb5kdc[15420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:06:22 dist-test-slave-n106 krb5kdc[15420](info): set up 2 sockets
Sep 24 16:06:22 dist-test-slave-n106 krb5kdc[15420](info): commencing operation
krb5kdc: starting...
W20260924 16:06:23.897550 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:06:23 dist-test-slave-n106 krb5kdc[15420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:06:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
I20260924 16:06:24.015887 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:35977
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:37539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:35977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:24.123878 15433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:24.124202 15433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:24.124286 15433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:24.128274 15433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:24.128373 15433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:24.128412 15433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:24.128475 15433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:24.128508 15433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:24.133746 15433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:37539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:35977
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:35977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:24.134814 15433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:24.135836 15433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:24.144637 15438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:24.144644 15441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:24.144655 15439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:24.144794 15433 server_base.cc:1034] running on GCE node
I20260924 16:06:24.145468 15433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:24.146562 15433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:24.147728 15433 hybrid_clock.cc:648] HybridClock initialized: now 1790265984147699 us; error 39 us; skew 500 ppm
Sep 24 16:06:24 dist-test-slave-n106 krb5kdc[15420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265984, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:24.150472 15433 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:06:24.151656 15433 webserver.cc:460] Webserver started at http://127.25.109.62:41373/ using document root <none> and password file <none>
I20260924 16:06:24.152272 15433 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:24.152364 15433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:24.152659 15433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:24.154783 15433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "ddd4273244bc409f93579f2df6e247d5"
format_stamp: "Formatted at 2026-09-24 16:06:24 on dist-test-slave-n106"
server_key: "deecebe11173d70a77f58032b03142ab"
server_key_iv: "05e7e495112f1df00e666a799e36879b"
server_key_version: "encryptionkey@0"
I20260924 16:06:24.155393 15433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "ddd4273244bc409f93579f2df6e247d5"
format_stamp: "Formatted at 2026-09-24 16:06:24 on dist-test-slave-n106"
server_key: "deecebe11173d70a77f58032b03142ab"
server_key_iv: "05e7e495112f1df00e666a799e36879b"
server_key_version: "encryptionkey@0"
I20260924 16:06:24.159247 15433 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 16:06:24.161743 15448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:24.162909 15433 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:06:24.163055 15433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "ddd4273244bc409f93579f2df6e247d5"
format_stamp: "Formatted at 2026-09-24 16:06:24 on dist-test-slave-n106"
server_key: "deecebe11173d70a77f58032b03142ab"
server_key_iv: "05e7e495112f1df00e666a799e36879b"
server_key_version: "encryptionkey@0"
I20260924 16:06:24.163168 15433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:24.182989 15433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:24.185918 15433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:24.186129 15433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:24.193156 15433 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:35977
I20260924 16:06:24.193224 15501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:35977 every 8 connection(s)
I20260924 16:06:24.194175 15433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:24.197000 15502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:24.202064 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 15433
I20260924 16:06:24.202216 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:06:24.202507 15502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddd4273244bc409f93579f2df6e247d5: Bootstrap starting.
I20260924 16:06:24.202533 26036 external_mini_cluster.cc:1428] Setting key f4c6c1cb3b59fd205ddfaa189a1b6881
I20260924 16:06:24.205291 15502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddd4273244bc409f93579f2df6e247d5: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:24.206159 15502 log.cc:824] T 00000000000000000000000000000000 P ddd4273244bc409f93579f2df6e247d5: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:24.208396 15502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddd4273244bc409f93579f2df6e247d5: No bootstrap required, opened a new log
Sep 24 16:06:24 dist-test-slave-n106 krb5kdc[15420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:06:24.212212 15502 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddd4273244bc409f93579f2df6e247d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddd4273244bc409f93579f2df6e247d5" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 35977 } }
I20260924 16:06:24.212488 15502 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddd4273244bc409f93579f2df6e247d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:24.212571 15502 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddd4273244bc409f93579f2df6e247d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddd4273244bc409f93579f2df6e247d5, State: Initialized, Role: FOLLOWER
I20260924 16:06:24.213096 15502 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddd4273244bc409f93579f2df6e247d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddd4273244bc409f93579f2df6e247d5" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 35977 } }
I20260924 16:06:24.213289 15502 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddd4273244bc409f93579f2df6e247d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:24.213366 15502 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddd4273244bc409f93579f2df6e247d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:24.213480 15502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddd4273244bc409f93579f2df6e247d5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:24.214491 15502 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddd4273244bc409f93579f2df6e247d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddd4273244bc409f93579f2df6e247d5" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 35977 } }
I20260924 16:06:24.214838 15502 leader_election.cc:302] T 00000000000000000000000000000000 P ddd4273244bc409f93579f2df6e247d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddd4273244bc409f93579f2df6e247d5; no voters:
I20260924 16:06:24.215170 15502 leader_election.cc:288] T 00000000000000000000000000000000 P ddd4273244bc409f93579f2df6e247d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:06:24.215276 15507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddd4273244bc409f93579f2df6e247d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:24.215495 15507 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddd4273244bc409f93579f2df6e247d5 [term 1 LEADER]: Becoming Leader. State: Replica: ddd4273244bc409f93579f2df6e247d5, State: Running, Role: LEADER
I20260924 16:06:24.215869 15507 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddd4273244bc409f93579f2df6e247d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddd4273244bc409f93579f2df6e247d5" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 35977 } }
I20260924 16:06:24.216584 15502 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddd4273244bc409f93579f2df6e247d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:24.217803 15508 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddd4273244bc409f93579f2df6e247d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddd4273244bc409f93579f2df6e247d5. Latest consensus state: current_term: 1 leader_uuid: "ddd4273244bc409f93579f2df6e247d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddd4273244bc409f93579f2df6e247d5" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 35977 } } }
I20260924 16:06:24.217926 15508 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddd4273244bc409f93579f2df6e247d5 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:24.218247 15505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:24.203932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36979 (local address 127.25.109.62:35977)
0924 16:06:24.204362 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:24.204381 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:24.204448 (+ 67us) server_negotiation.cc:407] Connection header received
0924 16:06:24.205325 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:24.205351 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:24.205811 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:24.206181 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:24.207039 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:24.208492 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:24.209348 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:24.210049 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:24.210215 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:24.212782 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:24.212787 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:24.212800 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:24.212828 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:24.214701 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:24.215293 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:24.215299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:24.215305 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:24.215506 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:24.215934 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:24.215937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:24.215938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:24.216369 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:24.216649 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:24.216919 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 16:06:24.217161 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":114,"threads_started":1}
I20260924 16:06:24.218216 15509 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddd4273244bc409f93579f2df6e247d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ddd4273244bc409f93579f2df6e247d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddd4273244bc409f93579f2df6e247d5" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 35977 } } }
I20260924 16:06:24.218611 15509 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddd4273244bc409f93579f2df6e247d5 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:24.218364 15516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:24.222363 26036 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:06:24.222381 15516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:24.227989 15516 catalog_manager.cc:1347] Generated new cluster ID: 267b21f926f2420aa2fa9a4565acc3ec
I20260924 16:06:24.228068 15516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:24.237403 15505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:24.224743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54195 (local address 127.25.109.62:35977)
0924 16:06:24.224849 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:24.224854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:24.228721 (+ 3867us) server_negotiation.cc:407] Connection header received
0924 16:06:24.228853 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:24.228857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:24.228903 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:24.228982 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:24.229830 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:24.230614 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:24.231786 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:24.232130 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:24.232220 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:24.233952 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:24.233956 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:24.233959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:24.233985 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:24.235588 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:24.236111 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:24.236114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:24.236116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:24.236195 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:24.236543 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:24.236546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:24.236548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:24.236741 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:24.236871 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:24.237158 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 16:06:24.237249 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 16:06:24.246016 15516 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:24.246982 15516 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:24.254021 15516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ddd4273244bc409f93579f2df6e247d5: Generated new TSK 0
I20260924 16:06:24.254817 15516 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:06:24.261629 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 15433
W20260924 16:06:24.269670 15418 connection.cc:535] client connection to 127.25.109.62:35977 recv error: Network error: failed to read from TLS socket (remote: 127.25.109.62:35977): Connection reset by peer (error 104)
2026-09-24T16:06:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:06:24 dist-test-slave-n106 krb5kdc[15534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:06:24 dist-test-slave-n106 krb5kdc[15534](info): set up 2 sockets
Sep 24 16:06:24 dist-test-slave-n106 krb5kdc[15534](info): commencing operation
krb5kdc: starting...
W20260924 16:06:25.297660 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:06:25 dist-test-slave-n106 krb5kdc[15534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:06:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
I20260924 16:06:25.418937 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:44253
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:41671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:44253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:25.528265 15547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:25.528590 15547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:25.528672 15547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:25.532606 15547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:25.532706 15547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:25.532743 15547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:25.532799 15547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:25.532837 15547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:25.538002 15547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:41671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:44253
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:44253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:25.539085 15547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:25.540122 15547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:25.548501 15555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:25.548501 15552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:25.548578 15553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:25.548755 15547 server_base.cc:1034] running on GCE node
I20260924 16:06:25.549386 15547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:25.550392 15547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:25.551584 15547 hybrid_clock.cc:648] HybridClock initialized: now 1790265985551551 us; error 41 us; skew 500 ppm
Sep 24 16:06:25 dist-test-slave-n106 krb5kdc[15534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265985, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:25.554231 15547 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:06:25.555111 15547 webserver.cc:460] Webserver started at http://127.25.109.62:46289/ using document root <none> and password file <none>
I20260924 16:06:25.555660 15547 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:25.555756 15547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:25.555996 15547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:25.558063 15547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "44c1f19619554b50ac158a76c085b97e"
format_stamp: "Formatted at 2026-09-24 16:06:25 on dist-test-slave-n106"
server_key: "ea56c729cdb0f401bd4de8c119463891"
server_key_iv: "adb773ab14e9e57ec6873d3f77398ea5"
server_key_version: "encryptionkey@0"
I20260924 16:06:25.558629 15547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "44c1f19619554b50ac158a76c085b97e"
format_stamp: "Formatted at 2026-09-24 16:06:25 on dist-test-slave-n106"
server_key: "ea56c729cdb0f401bd4de8c119463891"
server_key_iv: "adb773ab14e9e57ec6873d3f77398ea5"
server_key_version: "encryptionkey@0"
I20260924 16:06:25.562237 15547 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 16:06:25.564589 15562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:25.565810 15547 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 16:06:25.565907 15547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "44c1f19619554b50ac158a76c085b97e"
format_stamp: "Formatted at 2026-09-24 16:06:25 on dist-test-slave-n106"
server_key: "ea56c729cdb0f401bd4de8c119463891"
server_key_iv: "adb773ab14e9e57ec6873d3f77398ea5"
server_key_version: "encryptionkey@0"
I20260924 16:06:25.565985 15547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:25.576385 15547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:25.578989 15547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:25.579142 15547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:25.586278 15547 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:44253
I20260924 16:06:25.586305 15614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:44253 every 8 connection(s)
I20260924 16:06:25.587198 15547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:25.589890 15615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:25.594872 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 15547
I20260924 16:06:25.595045 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:06:25.595283 15615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44c1f19619554b50ac158a76c085b97e: Bootstrap starting.
I20260924 16:06:25.595327 26036 external_mini_cluster.cc:1428] Setting key c07ced03e79ade2b9767c2eb336c12bb
I20260924 16:06:25.597996 15615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44c1f19619554b50ac158a76c085b97e: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:25.598853 15615 log.cc:824] T 00000000000000000000000000000000 P 44c1f19619554b50ac158a76c085b97e: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:25.600991 15615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44c1f19619554b50ac158a76c085b97e: No bootstrap required, opened a new log
I20260924 16:06:25.605413 15618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:25.596705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45439 (local address 127.25.109.62:44253)
0924 16:06:25.597124 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:25.597143 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:25.597185 (+ 42us) server_negotiation.cc:407] Connection header received
0924 16:06:25.598015 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:25.598041 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:25.598489 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:25.598843 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:25.599283 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:25.600665 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:25.601599 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:25.602264 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:25.602436 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:25.603269 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:25.603275 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:25.603292 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:25.603874 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:25.603954 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:25.603994 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:25.604148 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:25.604291 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 16:06:25.604449 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":109,"threads_started":1}
I20260924 16:06:25.605383 15615 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44c1f19619554b50ac158a76c085b97e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c1f19619554b50ac158a76c085b97e" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44253 } }
I20260924 16:06:25.605726 15615 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44c1f19619554b50ac158a76c085b97e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:25.605805 15615 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44c1f19619554b50ac158a76c085b97e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c1f19619554b50ac158a76c085b97e, State: Initialized, Role: FOLLOWER
I20260924 16:06:25.606215 15615 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44c1f19619554b50ac158a76c085b97e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c1f19619554b50ac158a76c085b97e" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44253 } }
I20260924 16:06:25.606369 15615 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44c1f19619554b50ac158a76c085b97e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:25.606460 15615 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44c1f19619554b50ac158a76c085b97e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:25.606560 15615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44c1f19619554b50ac158a76c085b97e [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:25.607650 15615 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44c1f19619554b50ac158a76c085b97e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c1f19619554b50ac158a76c085b97e" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44253 } }
I20260924 16:06:25.608042 15615 leader_election.cc:302] T 00000000000000000000000000000000 P 44c1f19619554b50ac158a76c085b97e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44c1f19619554b50ac158a76c085b97e; no voters:
I20260924 16:06:25.608357 15615 leader_election.cc:288] T 00000000000000000000000000000000 P 44c1f19619554b50ac158a76c085b97e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:06:25.608474 15620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44c1f19619554b50ac158a76c085b97e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:25.608530 26036 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:06:25.608693 15620 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44c1f19619554b50ac158a76c085b97e [term 1 LEADER]: Becoming Leader. State: Replica: 44c1f19619554b50ac158a76c085b97e, State: Running, Role: LEADER
I20260924 16:06:25.608973 15620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44c1f19619554b50ac158a76c085b97e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c1f19619554b50ac158a76c085b97e" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44253 } }
I20260924 16:06:25.609578 15615 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44c1f19619554b50ac158a76c085b97e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:25.610229 15620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44c1f19619554b50ac158a76c085b97e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44c1f19619554b50ac158a76c085b97e. Latest consensus state: current_term: 1 leader_uuid: "44c1f19619554b50ac158a76c085b97e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c1f19619554b50ac158a76c085b97e" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44253 } } }
I20260924 16:06:25.610337 15620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44c1f19619554b50ac158a76c085b97e [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:25.610661 15621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44c1f19619554b50ac158a76c085b97e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44c1f19619554b50ac158a76c085b97e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c1f19619554b50ac158a76c085b97e" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44253 } } }
I20260924 16:06:25.610759 15621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44c1f19619554b50ac158a76c085b97e [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:25.610805 15631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:25.614118 15631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:25.615929 15618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:25.611090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49057 (local address 127.25.109.62:44253)
0924 16:06:25.611227 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:25.611233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:25.611353 (+ 120us) server_negotiation.cc:407] Connection header received
0924 16:06:25.611551 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:25.611554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:25.611607 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:25.611699 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:25.611977 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:25.612744 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:25.613912 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:25.614259 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:25.614364 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:25.615431 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:25.615435 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:25.615437 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:25.615456 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:25.615481 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:25.615490 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:25.615590 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:25.615749 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 16:06:25.615786 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 16:06:25.620462 15631 catalog_manager.cc:1347] Generated new cluster ID: ed57b9fe62cc41b8834e7103a204f7c6
I20260924 16:06:25.620594 15631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:25.642555 15631 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:25.643577 15631 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:25.649031 15631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44c1f19619554b50ac158a76c085b97e: Generated new TSK 0
I20260924 16:06:25.649891 15631 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:06:25.660482 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 15547
W20260924 16:06:25.666425 15532 connection.cc:535] client connection to 127.25.109.62:44253 recv error: Network error: failed to read from TLS socket (remote: 127.25.109.62:44253): Connection reset by peer (error 104)
2026-09-24T16:06:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1396 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:06:25 dist-test-slave-n106 krb5kdc[15646](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:06:25 dist-test-slave-n106 krb5kdc[15646](info): set up 2 sockets
Sep 24 16:06:25 dist-test-slave-n106 krb5kdc[15646](info): commencing operation
krb5kdc: starting...
W20260924 16:06:26.697721 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:06:26 dist-test-slave-n106 krb5kdc[15646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:06:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
I20260924 16:06:26.817327 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:46029
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:45315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:46029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:26.931260 15659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:26.931555 15659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:26.931634 15659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:26.935572 15659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:26.935668 15659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:26.935711 15659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:26.935748 15659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:26.935782 15659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:26.940968 15659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:45315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:46029
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:46029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:26.942072 15659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:26.943125 15659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:06:26.951622 15659 server_base.cc:1034] running on GCE node
W20260924 16:06:26.951640 15667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:26.951639 15664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:26.951822 15665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:26.952455 15659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:26.953568 15659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:26.954758 15659 hybrid_clock.cc:648] HybridClock initialized: now 1790265986954723 us; error 48 us; skew 500 ppm
Sep 24 16:06:26 dist-test-slave-n106 krb5kdc[15646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265986, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:26.962230 15659 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:06:26.963176 15659 webserver.cc:460] Webserver started at http://127.25.109.62:33195/ using document root <none> and password file <none>
I20260924 16:06:26.963765 15659 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:26.963856 15659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:26.964089 15659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:26.966188 15659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "a28861715381418682af7786d1730272"
format_stamp: "Formatted at 2026-09-24 16:06:26 on dist-test-slave-n106"
server_key: "ec1da1a712b544c07a65858df903f004"
server_key_iv: "feecdf817606a88df391d66f5ef00d31"
server_key_version: "encryptionkey@0"
I20260924 16:06:26.966777 15659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "a28861715381418682af7786d1730272"
format_stamp: "Formatted at 2026-09-24 16:06:26 on dist-test-slave-n106"
server_key: "ec1da1a712b544c07a65858df903f004"
server_key_iv: "feecdf817606a88df391d66f5ef00d31"
server_key_version: "encryptionkey@0"
I20260924 16:06:26.970517 15659 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 16:06:26.972913 15674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:26.974243 15659 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:06:26.974390 15659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "a28861715381418682af7786d1730272"
format_stamp: "Formatted at 2026-09-24 16:06:26 on dist-test-slave-n106"
server_key: "ec1da1a712b544c07a65858df903f004"
server_key_iv: "feecdf817606a88df391d66f5ef00d31"
server_key_version: "encryptionkey@0"
I20260924 16:06:26.974509 15659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:26.979097 15659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:26.982018 15659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:26.982226 15659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:26.989010 15659 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:46029
I20260924 16:06:26.989020 15727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:46029 every 8 connection(s)
I20260924 16:06:26.989890 15659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:26.992987 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 15659
I20260924 16:06:26.993048 15728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:26.993155 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:06:26.993477 26036 external_mini_cluster.cc:1428] Setting key c6378b8d389f6eea504fafa7d329da2e
I20260924 16:06:26.998797 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.994884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47587 (local address 127.25.109.62:46029)
0924 16:06:26.995309 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:26.995324 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:26.995357 (+ 33us) server_negotiation.cc:407] Connection header received
0924 16:06:26.996028 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:26.996045 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:26.996334 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:26.996702 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:26.997566 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:26.997572 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:26.997584 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:26.997595 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:26.997651 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:26.997671 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:26.997783 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:26.997903 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 16:06:26.998102 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":134,"threads_started":1}
I20260924 16:06:26.999768 15728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a28861715381418682af7786d1730272: Bootstrap starting.
I20260924 16:06:27.001542 26036 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:06:27.002357 15728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a28861715381418682af7786d1730272: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:27.003332 15728 log.cc:824] T 00000000000000000000000000000000 P a28861715381418682af7786d1730272: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:27.005501 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:27.003918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41797 (local address 127.25.109.62:46029)
0924 16:06:27.004052 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:27.004058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:27.004263 (+ 205us) server_negotiation.cc:407] Connection header received
0924 16:06:27.004342 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:27.004345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:27.004378 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:27.004477 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:27.005165 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:27.005169 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:27.005171 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:27.005183 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:27.005211 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:27.005223 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:27.005315 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:27.005350 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 16:06:27.005385 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:06:27.005764 15728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a28861715381418682af7786d1730272: No bootstrap required, opened a new log
I20260924 16:06:27.009461 15728 raft_consensus.cc:348] T 00000000000000000000000000000000 P a28861715381418682af7786d1730272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a28861715381418682af7786d1730272" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 46029 } }
I20260924 16:06:27.009672 15728 raft_consensus.cc:374] T 00000000000000000000000000000000 P a28861715381418682af7786d1730272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:27.009740 15728 raft_consensus.cc:729] T 00000000000000000000000000000000 P a28861715381418682af7786d1730272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a28861715381418682af7786d1730272, State: Initialized, Role: FOLLOWER
I20260924 16:06:27.010203 15728 consensus_queue.cc:260] T 00000000000000000000000000000000 P a28861715381418682af7786d1730272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a28861715381418682af7786d1730272" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 46029 } }
I20260924 16:06:27.010326 15728 raft_consensus.cc:388] T 00000000000000000000000000000000 P a28861715381418682af7786d1730272 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:27.010416 15728 raft_consensus.cc:482] T 00000000000000000000000000000000 P a28861715381418682af7786d1730272 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:27.010533 15728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a28861715381418682af7786d1730272 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:27.011420 15728 raft_consensus.cc:504] T 00000000000000000000000000000000 P a28861715381418682af7786d1730272 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a28861715381418682af7786d1730272" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 46029 } }
I20260924 16:06:27.011781 15728 leader_election.cc:302] T 00000000000000000000000000000000 P a28861715381418682af7786d1730272 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a28861715381418682af7786d1730272; no voters:
I20260924 16:06:27.012143 15728 leader_election.cc:288] T 00000000000000000000000000000000 P a28861715381418682af7786d1730272 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:06:27.012251 15739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a28861715381418682af7786d1730272 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:27.012475 15739 raft_consensus.cc:686] T 00000000000000000000000000000000 P a28861715381418682af7786d1730272 [term 1 LEADER]: Becoming Leader. State: Replica: a28861715381418682af7786d1730272, State: Running, Role: LEADER
I20260924 16:06:27.012859 15739 consensus_queue.cc:237] T 00000000000000000000000000000000 P a28861715381418682af7786d1730272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a28861715381418682af7786d1730272" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 46029 } }
I20260924 16:06:27.013365 15728 sys_catalog.cc:564] T 00000000000000000000000000000000 P a28861715381418682af7786d1730272 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:27.014636 15740 sys_catalog.cc:455] T 00000000000000000000000000000000 P a28861715381418682af7786d1730272 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a28861715381418682af7786d1730272" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a28861715381418682af7786d1730272" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 46029 } } }
I20260924 16:06:27.014642 15741 sys_catalog.cc:455] T 00000000000000000000000000000000 P a28861715381418682af7786d1730272 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a28861715381418682af7786d1730272. Latest consensus state: current_term: 1 leader_uuid: "a28861715381418682af7786d1730272" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a28861715381418682af7786d1730272" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 46029 } } }
I20260924 16:06:27.014763 15741 sys_catalog.cc:458] T 00000000000000000000000000000000 P a28861715381418682af7786d1730272 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:27.014762 15740 sys_catalog.cc:458] T 00000000000000000000000000000000 P a28861715381418682af7786d1730272 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:27.015239 15749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:27.018395 15749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:27.024102 15749 catalog_manager.cc:1347] Generated new cluster ID: c4f232015e55489dab7f4975f769d13d
I20260924 16:06:27.024187 15749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:27.040452 15749 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:27.040794 15749 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:27.049518 15749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a28861715381418682af7786d1730272: Generated new TSK 0
I20260924 16:06:27.050196 15749 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:06:27.056216 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 15659
2026-09-24T16:06:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1395 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:06:27 dist-test-slave-n106 krb5kdc[15761](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:06:27 dist-test-slave-n106 krb5kdc[15761](info): set up 2 sockets
Sep 24 16:06:27 dist-test-slave-n106 krb5kdc[15761](info): commencing operation
krb5kdc: starting...
W20260924 16:06:28.097595 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:06:28 dist-test-slave-n106 krb5kdc[15761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:06:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
I20260924 16:06:28.217291 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:41901
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:38621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:41901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:28.331879 15775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:28.332232 15775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:28.332319 15775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:28.336525 15775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:28.336622 15775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:28.336659 15775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:28.336696 15775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:28.336731 15775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:28.342207 15775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:38621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:41901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:41901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:28.343325 15775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:28.344497 15775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:28.353078 15781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:28.353080 15783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:28.353152 15775 server_base.cc:1034] running on GCE node
W20260924 16:06:28.353080 15780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:28.353840 15775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:28.354975 15775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:28.356139 15775 hybrid_clock.cc:648] HybridClock initialized: now 1790265988356106 us; error 44 us; skew 500 ppm
Sep 24 16:06:28 dist-test-slave-n106 krb5kdc[15761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265988, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:28.358765 15775 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:06:28.359596 15775 webserver.cc:460] Webserver started at http://127.25.109.62:45217/ using document root <none> and password file <none>
I20260924 16:06:28.360152 15775 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:28.360239 15775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:28.360523 15775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:28.362541 15775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "df07fde417de40b29d4a1e4e350e6eee"
format_stamp: "Formatted at 2026-09-24 16:06:28 on dist-test-slave-n106"
server_key: "d85ae4e15f208326b87d8e4ffcabe30d"
server_key_iv: "0745c4b19b9d957c4cdd99929f6f44e6"
server_key_version: "encryptionkey@0"
I20260924 16:06:28.363093 15775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "df07fde417de40b29d4a1e4e350e6eee"
format_stamp: "Formatted at 2026-09-24 16:06:28 on dist-test-slave-n106"
server_key: "d85ae4e15f208326b87d8e4ffcabe30d"
server_key_iv: "0745c4b19b9d957c4cdd99929f6f44e6"
server_key_version: "encryptionkey@0"
I20260924 16:06:28.366750 15775 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 16:06:28.369102 15790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:28.370280 15775 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:06:28.370417 15775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "df07fde417de40b29d4a1e4e350e6eee"
format_stamp: "Formatted at 2026-09-24 16:06:28 on dist-test-slave-n106"
server_key: "d85ae4e15f208326b87d8e4ffcabe30d"
server_key_iv: "0745c4b19b9d957c4cdd99929f6f44e6"
server_key_version: "encryptionkey@0"
I20260924 16:06:28.370527 15775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:28.387811 15775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:28.390588 15775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:28.390791 15775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:28.397473 15775 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:41901
I20260924 16:06:28.397498 15845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:41901 every 8 connection(s)
I20260924 16:06:28.398419 15775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:28.401211 15846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:28.403137 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 15775
I20260924 16:06:28.403285 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:06:28.403611 26036 external_mini_cluster.cc:1428] Setting key f270cecb750aa90c9257a465d681c927
I20260924 16:06:28.407766 15846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df07fde417de40b29d4a1e4e350e6eee: Bootstrap starting.
I20260924 16:06:28.410108 15846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df07fde417de40b29d4a1e4e350e6eee: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:28.410941 15846 log.cc:824] T 00000000000000000000000000000000 P df07fde417de40b29d4a1e4e350e6eee: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:06:28 dist-test-slave-n106 krb5kdc[15761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:06:28.413277 15846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df07fde417de40b29d4a1e4e350e6eee: No bootstrap required, opened a new log
I20260924 16:06:28.417485 15846 raft_consensus.cc:348] T 00000000000000000000000000000000 P df07fde417de40b29d4a1e4e350e6eee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df07fde417de40b29d4a1e4e350e6eee" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41901 } }
I20260924 16:06:28.417704 15846 raft_consensus.cc:374] T 00000000000000000000000000000000 P df07fde417de40b29d4a1e4e350e6eee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:28.417765 15846 raft_consensus.cc:729] T 00000000000000000000000000000000 P df07fde417de40b29d4a1e4e350e6eee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df07fde417de40b29d4a1e4e350e6eee, State: Initialized, Role: FOLLOWER
I20260924 16:06:28.417872 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:28.405078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50311 (local address 127.25.109.62:41901)
0924 16:06:28.405515 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:28.405530 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:28.405567 (+ 37us) server_negotiation.cc:407] Connection header received
0924 16:06:28.406231 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:28.406250 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:28.406581 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:28.406925 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:28.407734 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:28.408824 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:28.409696 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:28.410238 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:28.410404 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:28.413199 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:28.413204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:28.413216 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:28.413242 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:28.414954 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:28.415506 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:28.415511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:28.415515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:28.415627 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:28.415916 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:28.415918 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:28.415919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:28.416284 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:28.416497 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:28.416750 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 16:06:28.417050 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":129,"threads_started":1}
I20260924 16:06:28.418298 15846 consensus_queue.cc:260] T 00000000000000000000000000000000 P df07fde417de40b29d4a1e4e350e6eee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df07fde417de40b29d4a1e4e350e6eee" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41901 } }
I20260924 16:06:28.418452 15846 raft_consensus.cc:388] T 00000000000000000000000000000000 P df07fde417de40b29d4a1e4e350e6eee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:28.418540 15846 raft_consensus.cc:482] T 00000000000000000000000000000000 P df07fde417de40b29d4a1e4e350e6eee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:28.418665 15846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df07fde417de40b29d4a1e4e350e6eee [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:28.419775 15846 raft_consensus.cc:504] T 00000000000000000000000000000000 P df07fde417de40b29d4a1e4e350e6eee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df07fde417de40b29d4a1e4e350e6eee" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41901 } }
I20260924 16:06:28.420136 15846 leader_election.cc:302] T 00000000000000000000000000000000 P df07fde417de40b29d4a1e4e350e6eee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df07fde417de40b29d4a1e4e350e6eee; no voters:
I20260924 16:06:28.420516 15846 leader_election.cc:288] T 00000000000000000000000000000000 P df07fde417de40b29d4a1e4e350e6eee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:06:28.420666 15851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df07fde417de40b29d4a1e4e350e6eee [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:28.421038 26036 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:06:28.421082 15851 raft_consensus.cc:686] T 00000000000000000000000000000000 P df07fde417de40b29d4a1e4e350e6eee [term 1 LEADER]: Becoming Leader. State: Replica: df07fde417de40b29d4a1e4e350e6eee, State: Running, Role: LEADER
I20260924 16:06:28.421586 15851 consensus_queue.cc:237] T 00000000000000000000000000000000 P df07fde417de40b29d4a1e4e350e6eee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df07fde417de40b29d4a1e4e350e6eee" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41901 } }
I20260924 16:06:28.421857 15846 sys_catalog.cc:564] T 00000000000000000000000000000000 P df07fde417de40b29d4a1e4e350e6eee [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:28.423370 15851 sys_catalog.cc:455] T 00000000000000000000000000000000 P df07fde417de40b29d4a1e4e350e6eee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df07fde417de40b29d4a1e4e350e6eee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df07fde417de40b29d4a1e4e350e6eee" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41901 } } }
I20260924 16:06:28.423588 15851 sys_catalog.cc:458] T 00000000000000000000000000000000 P df07fde417de40b29d4a1e4e350e6eee [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:28.423856 15851 sys_catalog.cc:455] T 00000000000000000000000000000000 P df07fde417de40b29d4a1e4e350e6eee [sys.catalog]: SysCatalogTable state changed. Reason: New leader df07fde417de40b29d4a1e4e350e6eee. Latest consensus state: current_term: 1 leader_uuid: "df07fde417de40b29d4a1e4e350e6eee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df07fde417de40b29d4a1e4e350e6eee" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 41901 } } }
I20260924 16:06:28.424351 15869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:28.424356 15851 sys_catalog.cc:458] T 00000000000000000000000000000000 P df07fde417de40b29d4a1e4e350e6eee [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:28.427901 15869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:28.434129 15869 catalog_manager.cc:1347] Generated new cluster ID: 1757b343c45f466e99687c5a0017e3f6
I20260924 16:06:28.434264 15869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:28.435539 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:28.426116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47097 (local address 127.25.109.62:41901)
0924 16:06:28.426226 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:28.426230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:28.426832 (+ 602us) server_negotiation.cc:407] Connection header received
0924 16:06:28.426982 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:28.426986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:28.427032 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:28.427102 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:28.427982 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:28.428597 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:28.429600 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:28.429933 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:28.430042 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:28.431726 (+ 1684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:28.431730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:28.431733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:28.431760 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:28.433628 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:28.434319 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:28.434322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:28.434324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:28.434414 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:28.434693 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:28.434697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:28.434699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:28.434884 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:28.435015 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:28.435253 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 16:06:28.435365 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 16:06:28.465682 15869 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:28.466715 15869 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:28.474071 15869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df07fde417de40b29d4a1e4e350e6eee: Generated new TSK 0
I20260924 16:06:28.474845 15869 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:06:28.478782 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 15775
2026-09-24T16:06:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:06:28 dist-test-slave-n106 krb5kdc[15877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:06:28 dist-test-slave-n106 krb5kdc[15877](info): set up 2 sockets
Sep 24 16:06:28 dist-test-slave-n106 krb5kdc[15877](info): commencing operation
krb5kdc: starting...
W20260924 16:06:29.521708 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:06:29 dist-test-slave-n106 krb5kdc[15877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:06:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
I20260924 16:06:29.645604 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:43249
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:36583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:43249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:29.760826 15891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:29.761108 15891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:29.761152 15891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:29.765103 15891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:29.765162 15891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:29.765182 15891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:29.765197 15891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:29.765213 15891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:29.770426 15891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:36583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:43249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:43249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:29.771467 15891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:29.772500 15891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:29.780906 15896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:29.780925 15897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:29.781159 15891 server_base.cc:1034] running on GCE node
W20260924 16:06:29.780957 15899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:29.781888 15891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:29.783002 15891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:29.784216 15891 hybrid_clock.cc:648] HybridClock initialized: now 1790265989784200 us; error 28 us; skew 500 ppm
Sep 24 16:06:29 dist-test-slave-n106 krb5kdc[15877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265989, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:29.787077 15891 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:06:29.787937 15891 webserver.cc:460] Webserver started at http://127.25.109.62:36691/ using document root <none> and password file <none>
I20260924 16:06:29.788532 15891 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:29.788597 15891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:29.788769 15891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:29.790725 15891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "5fed14a92c5343958ff624a625901ae8"
format_stamp: "Formatted at 2026-09-24 16:06:29 on dist-test-slave-n106"
server_key: "a56a53a17dc7d368296c21609c592f56"
server_key_iv: "5f78f717d691fa71fa1e2bcd6347ab2b"
server_key_version: "encryptionkey@0"
I20260924 16:06:29.791240 15891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "5fed14a92c5343958ff624a625901ae8"
format_stamp: "Formatted at 2026-09-24 16:06:29 on dist-test-slave-n106"
server_key: "a56a53a17dc7d368296c21609c592f56"
server_key_iv: "5f78f717d691fa71fa1e2bcd6347ab2b"
server_key_version: "encryptionkey@0"
I20260924 16:06:29.794993 15891 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 16:06:29.797403 15906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:29.798669 15891 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:06:29.798827 15891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "5fed14a92c5343958ff624a625901ae8"
format_stamp: "Formatted at 2026-09-24 16:06:29 on dist-test-slave-n106"
server_key: "a56a53a17dc7d368296c21609c592f56"
server_key_iv: "5f78f717d691fa71fa1e2bcd6347ab2b"
server_key_version: "encryptionkey@0"
I20260924 16:06:29.798959 15891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:29.820582 15891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:29.824007 15891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:29.824226 15891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:29.831621 15891 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:43249
I20260924 16:06:29.831646 15959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:43249 every 8 connection(s)
I20260924 16:06:29.832757 15891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:29.835707 15960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:29.841454 15960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fed14a92c5343958ff624a625901ae8: Bootstrap starting.
I20260924 16:06:29.842370 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 15891
I20260924 16:06:29.842519 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:06:29.842820 26036 external_mini_cluster.cc:1428] Setting key 8f40798b57edf94203460b4ab673057c
I20260924 16:06:29.844489 15960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fed14a92c5343958ff624a625901ae8: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:29.845398 15960 log.cc:824] T 00000000000000000000000000000000 P 5fed14a92c5343958ff624a625901ae8: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:29.847625 15960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fed14a92c5343958ff624a625901ae8: No bootstrap required, opened a new log
I20260924 16:06:29.851974 15960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fed14a92c5343958ff624a625901ae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fed14a92c5343958ff624a625901ae8" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 43249 } }
I20260924 16:06:29.852253 15960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fed14a92c5343958ff624a625901ae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:29.852340 15960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fed14a92c5343958ff624a625901ae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fed14a92c5343958ff624a625901ae8, State: Initialized, Role: FOLLOWER
I20260924 16:06:29.852880 15960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fed14a92c5343958ff624a625901ae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fed14a92c5343958ff624a625901ae8" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 43249 } }
I20260924 16:06:29.853034 15963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:29.844038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55369 (local address 127.25.109.62:43249)
0924 16:06:29.844573 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:29.844595 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:29.844640 (+ 45us) server_negotiation.cc:407] Connection header received
0924 16:06:29.845528 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:29.845553 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:29.845978 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:29.846354 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:29.846746 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:29.848174 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:29.849116 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:29.849830 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:29.849995 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:29.850814 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:29.850821 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:29.850838 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:29.851377 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:29.851449 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:29.851481 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:29.851584 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:29.851722 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 16:06:29.851937 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":137,"threads_started":1}
I20260924 16:06:29.853048 15960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fed14a92c5343958ff624a625901ae8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:29.853189 15960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fed14a92c5343958ff624a625901ae8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:29.853279 15960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fed14a92c5343958ff624a625901ae8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:29.854288 15960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fed14a92c5343958ff624a625901ae8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fed14a92c5343958ff624a625901ae8" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 43249 } }
I20260924 16:06:29.854665 15960 leader_election.cc:302] T 00000000000000000000000000000000 P 5fed14a92c5343958ff624a625901ae8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fed14a92c5343958ff624a625901ae8; no voters:
I20260924 16:06:29.855023 15960 leader_election.cc:288] T 00000000000000000000000000000000 P 5fed14a92c5343958ff624a625901ae8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:06:29.855108 15965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fed14a92c5343958ff624a625901ae8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:29.855368 15965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fed14a92c5343958ff624a625901ae8 [term 1 LEADER]: Becoming Leader. State: Replica: 5fed14a92c5343958ff624a625901ae8, State: Running, Role: LEADER
I20260924 16:06:29.855720 15965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fed14a92c5343958ff624a625901ae8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fed14a92c5343958ff624a625901ae8" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 43249 } }
I20260924 16:06:29.856129 26036 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:06:29.856489 15960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fed14a92c5343958ff624a625901ae8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:29.857568 15966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fed14a92c5343958ff624a625901ae8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fed14a92c5343958ff624a625901ae8. Latest consensus state: current_term: 1 leader_uuid: "5fed14a92c5343958ff624a625901ae8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fed14a92c5343958ff624a625901ae8" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 43249 } } }
I20260924 16:06:29.858007 15967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fed14a92c5343958ff624a625901ae8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fed14a92c5343958ff624a625901ae8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fed14a92c5343958ff624a625901ae8" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 43249 } } }
I20260924 16:06:29.858093 15967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fed14a92c5343958ff624a625901ae8 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:29.857743 15966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fed14a92c5343958ff624a625901ae8 [sys.catalog]: This master's current role is: LEADER
W20260924 16:06:29.862023 15985 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5fed14a92c5343958ff624a625901ae8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:06:29.862113 15985 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:06:29.862449 15979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:29.863384 15979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:29.863956 15963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:29.858972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34887 (local address 127.25.109.62:43249)
0924 16:06:29.859105 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:29.859111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:29.859313 (+ 202us) server_negotiation.cc:407] Connection header received
0924 16:06:29.859549 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:29.859555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:29.859612 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:29.859698 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:29.860009 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:29.860779 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:29.861951 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:29.862283 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:29.862368 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:29.863278 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:29.863284 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:29.863287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:29.863300 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:29.863332 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:29.863345 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:29.863430 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:29.863762 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 16:06:29.863818 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:06:29.870016 15979 catalog_manager.cc:1347] Generated new cluster ID: 00e863ad64224049b9972232e3ec9fb8
I20260924 16:06:29.870090 15979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:29.893560 15979 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:29.894955 15979 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:29.904901 15979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fed14a92c5343958ff624a625901ae8: Generated new TSK 0
I20260924 16:06:29.905795 15979 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:06:29.913529 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 15891
2026-09-24T16:06:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1434 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:06:29 dist-test-slave-n106 krb5kdc[15991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:06:29 dist-test-slave-n106 krb5kdc[15991](info): set up 2 sockets
Sep 24 16:06:29 dist-test-slave-n106 krb5kdc[15991](info): commencing operation
krb5kdc: starting...
W20260924 16:06:30.949681 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:06:31 dist-test-slave-n106 krb5kdc[15991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:06:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
I20260924 16:06:31.072925 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:42751
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:35149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:42751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:31.183380 16004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:31.183743 16004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:31.183835 16004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:31.187841 16004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:31.187938 16004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:31.187974 16004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:31.188012 16004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:31.188045 16004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:31.193308 16004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:35149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:42751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:42751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:31.194432 16004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:31.195611 16004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:31.204160 16010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:31.204211 16009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:31.204453 16004 server_base.cc:1034] running on GCE node
W20260924 16:06:31.204166 16012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:31.205156 16004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:31.206203 16004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:31.207362 16004 hybrid_clock.cc:648] HybridClock initialized: now 1790265991207344 us; error 31 us; skew 500 ppm
Sep 24 16:06:31 dist-test-slave-n106 krb5kdc[15991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265991, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:31.210095 16004 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:06:31.210974 16004 webserver.cc:460] Webserver started at http://127.25.109.62:35569/ using document root <none> and password file <none>
I20260924 16:06:31.211565 16004 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:31.211632 16004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:31.211902 16004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:31.214084 16004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "5edc55151d60497da853ec8a24215a58"
format_stamp: "Formatted at 2026-09-24 16:06:31 on dist-test-slave-n106"
server_key: "9754ca66196346f01c3cd3219c85c482"
server_key_iv: "6a7966773a603e4786d8bd2c791eea85"
server_key_version: "encryptionkey@0"
I20260924 16:06:31.214648 16004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "5edc55151d60497da853ec8a24215a58"
format_stamp: "Formatted at 2026-09-24 16:06:31 on dist-test-slave-n106"
server_key: "9754ca66196346f01c3cd3219c85c482"
server_key_iv: "6a7966773a603e4786d8bd2c791eea85"
server_key_version: "encryptionkey@0"
I20260924 16:06:31.218446 16004 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:06:31.220881 16019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:31.222105 16004 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:06:31.222239 16004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "5edc55151d60497da853ec8a24215a58"
format_stamp: "Formatted at 2026-09-24 16:06:31 on dist-test-slave-n106"
server_key: "9754ca66196346f01c3cd3219c85c482"
server_key_iv: "6a7966773a603e4786d8bd2c791eea85"
server_key_version: "encryptionkey@0"
I20260924 16:06:31.222340 16004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:31.226943 16004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:31.229782 16004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:31.229974 16004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:31.236757 16004 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:42751
I20260924 16:06:31.236783 16073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:42751 every 8 connection(s)
I20260924 16:06:31.237706 16004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:31.239104 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 16004
I20260924 16:06:31.239251 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:06:31.239532 26036 external_mini_cluster.cc:1428] Setting key bd7ee04c33496cda3616f90bb6afeea8
I20260924 16:06:31.240873 16074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:31.245463 16077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:31.241112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33349 (local address 127.25.109.62:42751)
0924 16:06:31.241546 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:31.241565 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:31.241618 (+ 53us) server_negotiation.cc:407] Connection header received
0924 16:06:31.242464 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:31.242486 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:31.242861 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:31.243236 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:31.244004 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:31.244010 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:31.244026 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:31.244039 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:31.244103 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:31.244134 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:31.244278 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:31.244349 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 16:06:31.244531 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":119,"threads_started":1}
I20260924 16:06:31.248034 16074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5edc55151d60497da853ec8a24215a58: Bootstrap starting.
I20260924 16:06:31.248426 26036 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:06:31.250331 16074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5edc55151d60497da853ec8a24215a58: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:31.251225 16074 log.cc:824] T 00000000000000000000000000000000 P 5edc55151d60497da853ec8a24215a58: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:31.252100 16077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:31.250470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38569 (local address 127.25.109.62:42751)
0924 16:06:31.250614 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:31.250619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:31.250772 (+ 153us) server_negotiation.cc:407] Connection header received
0924 16:06:31.250865 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:31.250868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:31.250900 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:31.250981 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:31.251680 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:31.251684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:31.251686 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:31.251698 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:31.251726 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:31.251744 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:31.251835 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:31.251938 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 16:06:31.251976 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 16:06:31.253333 16074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5edc55151d60497da853ec8a24215a58: No bootstrap required, opened a new log
I20260924 16:06:31.256721 16074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5edc55151d60497da853ec8a24215a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5edc55151d60497da853ec8a24215a58" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 42751 } }
I20260924 16:06:31.256953 16074 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5edc55151d60497da853ec8a24215a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:31.257040 16074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5edc55151d60497da853ec8a24215a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5edc55151d60497da853ec8a24215a58, State: Initialized, Role: FOLLOWER
I20260924 16:06:31.257491 16074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5edc55151d60497da853ec8a24215a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5edc55151d60497da853ec8a24215a58" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 42751 } }
I20260924 16:06:31.257643 16074 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5edc55151d60497da853ec8a24215a58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:31.257715 16074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5edc55151d60497da853ec8a24215a58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:31.257833 16074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5edc55151d60497da853ec8a24215a58 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:31.258662 16074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5edc55151d60497da853ec8a24215a58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5edc55151d60497da853ec8a24215a58" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 42751 } }
I20260924 16:06:31.258987 16074 leader_election.cc:302] T 00000000000000000000000000000000 P 5edc55151d60497da853ec8a24215a58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5edc55151d60497da853ec8a24215a58; no voters:
I20260924 16:06:31.259310 16074 leader_election.cc:288] T 00000000000000000000000000000000 P 5edc55151d60497da853ec8a24215a58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:06:31.259389 16084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5edc55151d60497da853ec8a24215a58 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:31.259677 16084 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5edc55151d60497da853ec8a24215a58 [term 1 LEADER]: Becoming Leader. State: Replica: 5edc55151d60497da853ec8a24215a58, State: Running, Role: LEADER
I20260924 16:06:31.260026 16084 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5edc55151d60497da853ec8a24215a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5edc55151d60497da853ec8a24215a58" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 42751 } }
I20260924 16:06:31.260604 16074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5edc55151d60497da853ec8a24215a58 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:31.261838 16085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5edc55151d60497da853ec8a24215a58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5edc55151d60497da853ec8a24215a58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5edc55151d60497da853ec8a24215a58" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 42751 } } }
I20260924 16:06:31.261982 16085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5edc55151d60497da853ec8a24215a58 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:31.262363 16086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5edc55151d60497da853ec8a24215a58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5edc55151d60497da853ec8a24215a58. Latest consensus state: current_term: 1 leader_uuid: "5edc55151d60497da853ec8a24215a58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5edc55151d60497da853ec8a24215a58" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 42751 } } }
I20260924 16:06:31.262465 16086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5edc55151d60497da853ec8a24215a58 [sys.catalog]: This master's current role is: LEADER
W20260924 16:06:31.265376 16099 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5edc55151d60497da853ec8a24215a58: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:06:31.265466 16099 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:06:31.265597 16094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:31.266484 16094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:31.271973 16094 catalog_manager.cc:1347] Generated new cluster ID: 23c073bce2934b56b07a9353d73055cb
I20260924 16:06:31.272047 16094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:31.294243 16094 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:31.294562 16094 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:31.301803 16094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5edc55151d60497da853ec8a24215a58: Generated new TSK 0
I20260924 16:06:31.302611 16094 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:06:31.314428 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 16004
2026-09-24T16:06:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1401 ms)
[----------] 18 tests from AuthTokenIssuingTest (25524 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:06:31 dist-test-slave-n106 krb5kdc[16105](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:06:31 dist-test-slave-n106 krb5kdc[16105](info): set up 2 sockets
Sep 24 16:06:31 dist-test-slave-n106 krb5kdc[16105](info): commencing operation
krb5kdc: starting...
W20260924 16:06:32.357782 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:06:32 dist-test-slave-n106 krb5kdc[16105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:06:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
I20260924 16:06:32.477695 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:32857
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:32969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:32857,127.25.109.61:37063,127.25.109.60:37807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:32.586887 16118 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 16:06:32.587154 16118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:32.587177 16118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:32.587255 16118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:32.591264 16118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:32.591315 16118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:32.591333 16118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:32.591351 16118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:32.591394 16118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:32.596589 16118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:32969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:32857,127.25.109.61:37063,127.25.109.60:37807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:32857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:32.597564 16118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:32.598556 16118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:32.606772 16126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:32.606781 16123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:32.606781 16124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:32.607098 16118 server_base.cc:1034] running on GCE node
I20260924 16:06:32.607586 16118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:32.608651 16118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:32.609822 16118 hybrid_clock.cc:648] HybridClock initialized: now 1790265992609804 us; error 29 us; skew 500 ppm
Sep 24 16:06:32 dist-test-slave-n106 krb5kdc[16105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265992, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:32.615787 16118 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:06:32.616674 16118 webserver.cc:460] Webserver started at http://127.25.109.62:40537/ using document root <none> and password file <none>
I20260924 16:06:32.617208 16118 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:32.617296 16118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:32.617516 16118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:32.619477 16118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "d2b6687955c34b4083cdc1c9e2de46d6"
format_stamp: "Formatted at 2026-09-24 16:06:32 on dist-test-slave-n106"
server_key: "f467c716e1875da56194bf2a6222de77"
server_key_iv: "9c4bf65dd82957d07640c4de1f30d55b"
server_key_version: "encryptionkey@0"
I20260924 16:06:32.620028 16118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "d2b6687955c34b4083cdc1c9e2de46d6"
format_stamp: "Formatted at 2026-09-24 16:06:32 on dist-test-slave-n106"
server_key: "f467c716e1875da56194bf2a6222de77"
server_key_iv: "9c4bf65dd82957d07640c4de1f30d55b"
server_key_version: "encryptionkey@0"
I20260924 16:06:32.623641 16118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 16:06:32.626078 16133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:32.627251 16118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:06:32.627389 16118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "d2b6687955c34b4083cdc1c9e2de46d6"
format_stamp: "Formatted at 2026-09-24 16:06:32 on dist-test-slave-n106"
server_key: "f467c716e1875da56194bf2a6222de77"
server_key_iv: "9c4bf65dd82957d07640c4de1f30d55b"
server_key_version: "encryptionkey@0"
I20260924 16:06:32.627499 16118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:32.642918 16118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:32.645779 16118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:32.645972 16118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:32.652705 16118 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:32857
I20260924 16:06:32.653007 16186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:32857 every 8 connection(s)
I20260924 16:06:32.653702 16118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:32.656383 16187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:32.660027 16187 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32857 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:06:32.663667 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 16118
I20260924 16:06:32.663797 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:06:32.664113 26036 external_mini_cluster.cc:1428] Setting key de4ded3ccbad778f4bbe95004808f45d
Sep 24 16:06:32 dist-test-slave-n106 krb5kdc[16105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265992, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.109.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.61@KRBTEST.COM" created.
I20260924 16:06:32.675675 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:32.662880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42918 (local address 127.25.109.62:32857)
0924 16:06:32.663151 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:32.663164 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:32.663198 (+ 34us) server_negotiation.cc:407] Connection header received
0924 16:06:32.663715 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:32.663724 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:32.663961 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:32.664154 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:32.665013 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:32.666215 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:32.667737 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:32.668091 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:32.668202 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:32.670798 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:32.670804 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:32.670812 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:32.670832 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:32.672500 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:32.672948 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:32.672953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:32.672955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:32.673060 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:32.673436 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:32.673439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:32.673441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:32.673779 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:32.673913 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:32.674353 (+ 440us) server_negotiation.cc:299] Negotiation successful
0924 16:06:32.674642 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":160,"thread_start_us":96,"threads_started":1}
I20260924 16:06:32.675673 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:32.662184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:32857 (local address 127.0.0.1:42918)
0924 16:06:32.662602 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:32.662641 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:32.663406 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:32.664185 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:32.664193 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:32.664616 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:32.664860 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:32.664872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:32.666352 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:32.666358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:32.667614 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:32.667629 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:32.668251 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:32.668259 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:32.668394 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:32.668997 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:32.669023 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:32.670653 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:32.672600 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:32.672617 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:32.672626 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:32.672839 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:32.673192 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:32.673195 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:32.673197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:32.673340 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:32.673899 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:32.673905 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:32.674038 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 16:06:32.674282 (+ 244us) client_negotiation.cc:240] Negotiation successful
0924 16:06:32.674689 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":118,"thread_start_us":114,"threads_started":1}
I20260924 16:06:32.678416 16187 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.61" port: 37063 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:06:32.679143 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:32.678890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:37063 (local address 127.0.0.1:54068)
0924 16:06:32.679004 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:32.679081 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.61:37063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 16:06:32.679392 16135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.61:37063: connect: Connection refused (error 111)
I20260924 16:06:32.680330 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:32.680049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:37063 (local address 127.0.0.1:54072)
0924 16:06:32.680179 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:32.680243 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.61:37063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 16:06:32.680505 16187 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.109.61:37063: Network error: Client connection negotiation failed: client connection to 127.25.109.61:37063: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.61.keytab.
Entry for principal kudu/127.25.109.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.61.keytab.
I20260924 16:06:32.689771 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.61:37063
--webserver_interface=127.25.109.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.61.keytab
--principal=kudu/127.25.109.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:32969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:32857,127.25.109.61:37063,127.25.109.60:37807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:06:32.729806 16187 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.109.61" port: 37063 } attempt: 1
I20260924 16:06:32.730844 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:32.730485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:37063 (local address 127.0.0.1:54082)
0924 16:06:32.730648 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:32.730717 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.61:37063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 16:06:32.731644 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:32.731396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:37063 (local address 127.0.0.1:54084)
0924 16:06:32.731540 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:32.731596 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.61:37063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260924 16:06:32.731792 16187 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.109.61:37063: Network error: Client connection negotiation failed: client connection to 127.25.109.61:37063: connect: Connection refused (error 111)
W20260924 16:06:32.799551 16194 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 16:06:32.799815 16194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:32.799837 16194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:32.799916 16194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 16:06:32.799998 16187 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.109.61" port: 37063 } attempt: 2
I20260924 16:06:32.800913 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:32.800593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:37063 (local address 127.0.0.1:54088)
0924 16:06:32.800751 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:32.800834 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.61:37063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 16:06:32.801630 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:32.801450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:37063 (local address 127.0.0.1:54090)
0924 16:06:32.801540 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:32.801583 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.61:37063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 16:06:32.801781 16187 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.109.61:37063: Network error: Client connection negotiation failed: client connection to 127.25.109.61:37063: connect: Connection refused (error 111)
W20260924 16:06:32.804817 16194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:32.804863 16194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:32.804883 16194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:32.804899 16194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:32.804915 16194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:32.810034 16194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:32969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:32857,127.25.109.61:37063,127.25.109.60:37807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.61.keytab
--principal=kudu/127.25.109.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.61:37063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.109.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:32.810993 16194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:32.811923 16194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:32.820190 16202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:32.820235 16200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:32.820361 16194 server_base.cc:1034] running on GCE node
W20260924 16:06:32.820235 16199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:32.820897 16194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:32.821918 16194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:32.823105 16194 hybrid_clock.cc:648] HybridClock initialized: now 1790265992823089 us; error 26 us; skew 500 ppm
Sep 24 16:06:32 dist-test-slave-n106 krb5kdc[16105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265992, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:32.825811 16194 init.cc:377] Logged in from keytab as kudu/127.25.109.61@KRBTEST.COM (short username kudu)
I20260924 16:06:32.826558 16194 webserver.cc:460] Webserver started at http://127.25.109.61:38169/ using document root <none> and password file <none>
I20260924 16:06:32.827044 16194 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:32.827097 16194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:32.827252 16194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:32.829196 16194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-1/data/instance:
uuid: "55bc470e70254d7dbfbee77e12012007"
format_stamp: "Formatted at 2026-09-24 16:06:32 on dist-test-slave-n106"
server_key: "acc761d5620141b0b989089a23c6c40b"
server_key_iv: "0a8ddf9b1ee2a9e4a09244240deba17c"
server_key_version: "encryptionkey@0"
I20260924 16:06:32.829695 16194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-1/wal/instance:
uuid: "55bc470e70254d7dbfbee77e12012007"
format_stamp: "Formatted at 2026-09-24 16:06:32 on dist-test-slave-n106"
server_key: "acc761d5620141b0b989089a23c6c40b"
server_key_iv: "0a8ddf9b1ee2a9e4a09244240deba17c"
server_key_version: "encryptionkey@0"
I20260924 16:06:32.833268 16194 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:06:32.835544 16210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:32.836822 16194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:06:32.836951 16194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-1/wal
uuid: "55bc470e70254d7dbfbee77e12012007"
format_stamp: "Formatted at 2026-09-24 16:06:32 on dist-test-slave-n106"
server_key: "acc761d5620141b0b989089a23c6c40b"
server_key_iv: "0a8ddf9b1ee2a9e4a09244240deba17c"
server_key_version: "encryptionkey@0"
I20260924 16:06:32.837059 16194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:32.854876 16194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:32.858165 16194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:32.858317 16194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:32.865031 16194 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.61:37063
I20260924 16:06:32.865312 16262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.61:37063 every 8 connection(s)
I20260924 16:06:32.866091 16194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-1/data/info.pb
I20260924 16:06:32.868878 16263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:32.872561 16263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32857 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:06:32.875706 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 16194
I20260924 16:06:32.875836 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-1/wal/instance
I20260924 16:06:32.876092 26036 external_mini_cluster.cc:1428] Setting key 86ed4bff482b6b9a93a322b009ecee21
Sep 24 16:06:32 dist-test-slave-n106 krb5kdc[16105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265992, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.61@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.109.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.60@KRBTEST.COM" created.
I20260924 16:06:32.887447 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:32.874399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42924 (local address 127.25.109.62:32857)
0924 16:06:32.874540 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:32.874543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:32.875328 (+ 785us) server_negotiation.cc:407] Connection header received
0924 16:06:32.876313 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:32.876318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:32.876381 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:32.876496 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:32.877750 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:32.878619 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:32.880455 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:32.880761 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:32.880847 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:32.883668 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:32.883671 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:32.883673 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:32.883695 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:32.885246 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:32.885755 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:32.885759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:32.885761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:32.885839 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:32.886179 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:32.886184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:32.886186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:32.886373 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:32.886474 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:32.887090 (+ 616us) server_negotiation.cc:299] Negotiation successful
0924 16:06:32.887278 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 16:06:32.888358 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:32.874750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:32857 (local address 127.0.0.1:42924)
0924 16:06:32.875176 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:32.875218 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:32.876074 (+ 856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:32.876694 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:32.876702 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:32.877023 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:32.877542 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:32.877564 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:32.878809 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:32.878815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:32.880277 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:32.880292 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:32.880931 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:32.880935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:32.881163 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:32.881818 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:32.881846 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:32.883490 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:32.885362 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:32.885368 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:32.885382 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:32.885665 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:32.885940 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:32.885943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:32.885945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:32.886081 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:32.886489 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:32.886496 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:32.886836 (+ 340us) client_negotiation.cc:769] Sending connection context
0924 16:06:32.887020 (+ 184us) client_negotiation.cc:240] Negotiation successful
0924 16:06:32.887333 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":116,"threads_started":1}
I20260924 16:06:32.889878 16263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.61" port: 37063 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:06:32.892941 16187 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.109.61" port: 37063 } attempt: 3
Sep 24 16:06:32 dist-test-slave-n106 krb5kdc[16105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265992, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.61@KRBTEST.COM for kudu/127.25.109.61@KRBTEST.COM
Sep 24 16:06:32 dist-test-slave-n106 krb5kdc[16105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265992, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for kudu/127.25.109.61@KRBTEST.COM
I20260924 16:06:32.901062 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:32.890540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54102 (local address 127.25.109.61:37063)
0924 16:06:32.890926 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:32.890944 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:32.890970 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:06:32.891017 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:32.891025 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:32.891229 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:32.891386 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:32.891724 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:32.893027 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:32.894005 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:32.894604 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:32.894729 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:32.896721 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:32.896729 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:32.896740 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:32.896759 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:32.898905 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:32.899399 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:32.899408 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:32.899410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:32.899515 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:32.899920 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:32.899927 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:32.899929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:32.900208 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:32.900481 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:32.900713 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 16:06:32.900836 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":267,"spinlock_wait_cycles":768,"thread_start_us":164,"threads_started":1}
I20260924 16:06:32.900972 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:32.890376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:37063 (local address 127.0.0.1:54102)
0924 16:06:32.890493 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:32.890507 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:32.890578 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:32.891384 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:32.891386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:32.891511 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:32.891599 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:32.891615 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:32.893144 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:32.893148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:32.893858 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:32.893889 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:32.894729 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:32.894732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:32.894897 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:32.895378 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:32.895390 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:32.896567 (+ 1177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:32.899042 (+ 2475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:32.899048 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:32.899050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:32.899290 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:32.899611 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:32.899614 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:32.899616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:32.899712 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:32.900334 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:32.900337 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:32.900491 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 16:06:32.900631 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 16:06:32.900770 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":59}
I20260924 16:06:32.903100 16263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.60" port: 37807 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:06:32.903481 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:32.893316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:37063 (local address 127.0.0.1:54106)
0924 16:06:32.893470 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:32.893484 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:32.893551 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:32.893940 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:32.893946 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:32.894117 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:32.894203 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:32.894208 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:32.895035 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:32.895039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:32.896533 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:32.896544 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:32.897172 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:32.897177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:32.897289 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:32.897671 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:32.897686 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:32.899305 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:32.901791 (+ 2486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:32.901795 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:32.901797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:32.902060 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:32.902405 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:32.902411 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:32.902415 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:32.902506 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:32.902946 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:32.902949 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:32.903055 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 16:06:32.903167 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 16:06:32.903304 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260924 16:06:32.903690 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:32.893405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54106 (local address 127.25.109.61:37063)
0924 16:06:32.893698 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:32.893707 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:32.893721 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:06:32.893757 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:32.893760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:32.893804 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:32.893902 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:32.894315 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:32.894916 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:32.896666 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:32.896957 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:32.897157 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:32.899462 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:32.899465 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:32.899466 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:32.899486 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:32.901689 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:32.902216 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:32.902222 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:32.902225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:32.902310 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:32.902601 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:32.902605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:32.902606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:32.902835 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:32.902955 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:32.903220 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 16:06:32.903323 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":102,"threads_started":1}
I20260924 16:06:32.903909 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:32.903613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:37807 (local address 127.0.0.1:40488)
0924 16:06:32.903735 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:32.903832 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:37807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 16:06:32.904196 16213 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.60:37807: connect: Connection refused (error 111)
I20260924 16:06:32.904433 16187 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.60" port: 37807 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.60.keytab.
Entry for principal kudu/127.25.109.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.60.keytab.
I20260924 16:06:32.905133 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:32.904902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:37807 (local address 127.0.0.1:40508)
0924 16:06:32.905026 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:32.905082 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:37807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 16:06:32.905223 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:32.904855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:37807 (local address 127.0.0.1:40496)
0924 16:06:32.905088 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:32.905145 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:37807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260924 16:06:32.905639 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.60:37807
--webserver_interface=127.25.109.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.60.keytab
--principal=kudu/127.25.109.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:32969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:32857,127.25.109.61:37063,127.25.109.60:37807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:06:32.906226 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:32.905732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:37807 (local address 127.0.0.1:40524)
0924 16:06:32.906079 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:32.906144 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:37807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260}
W20260924 16:06:32.906369 16187 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.109.60:37807: Network error: Client connection negotiation failed: client connection to 127.25.109.60:37807: connect: Connection refused (error 111)
W20260924 16:06:32.907094 16263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.109.60:37807: Network error: Client connection negotiation failed: client connection to 127.25.109.60:37807: connect: Connection refused (error 111)
I20260924 16:06:32.937543 16187 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.109.60" port: 37807 } attempt: 1
I20260924 16:06:32.938434 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:32.938082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:37807 (local address 127.0.0.1:40532)
0924 16:06:32.938261 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:32.938334 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:37807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 16:06:32.939180 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:32.938958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:37807 (local address 127.0.0.1:40546)
0924 16:06:32.939058 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:32.939111 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:37807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 16:06:32.939328 16187 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.109.60:37807: Network error: Client connection negotiation failed: client connection to 127.25.109.60:37807: connect: Connection refused (error 111)
I20260924 16:06:32.956513 16263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.109.60" port: 37807 } attempt: 1
I20260924 16:06:32.957399 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:32.957110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:37807 (local address 127.0.0.1:40552)
0924 16:06:32.957258 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:32.957321 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:37807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 16:06:32.958123 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:32.957940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:37807 (local address 127.0.0.1:40568)
0924 16:06:32.958032 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:32.958074 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:37807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 16:06:32.958261 16263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.109.60:37807: Network error: Client connection negotiation failed: client connection to 127.25.109.60:37807: connect: Connection refused (error 111)
I20260924 16:06:33.014511 16187 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.109.60" port: 37807 } attempt: 2
I20260924 16:06:33.015421 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.015066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:37807 (local address 127.0.0.1:40574)
0924 16:06:33.015255 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:33.015328 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:37807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 16:06:33.016170 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.015944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:37807 (local address 127.0.0.1:40582)
0924 16:06:33.016052 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:33.016105 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:37807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 16:06:33.016345 16187 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.109.60:37807: Network error: Client connection negotiation failed: client connection to 127.25.109.60:37807: connect: Connection refused (error 111)
W20260924 16:06:33.020265 16271 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 16:06:33.020605 16271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:33.020674 16271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:33.020714 16271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:33.024753 16271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:33.024798 16271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:33.024817 16271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:33.024868 16271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:33.024901 16271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:33.026429 16263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.109.60" port: 37807 } attempt: 2
I20260924 16:06:33.027168 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.026916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:37807 (local address 127.0.0.1:40588)
0924 16:06:33.027042 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:33.027096 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:37807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 16:06:33.027856 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.027675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:37807 (local address 127.0.0.1:40598)
0924 16:06:33.027766 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:33.027807 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:37807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 16:06:33.028043 16263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.109.60:37807: Network error: Client connection negotiation failed: client connection to 127.25.109.60:37807: connect: Connection refused (error 111)
I20260924 16:06:33.030792 16271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:32969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:32857,127.25.109.61:37063,127.25.109.60:37807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.60.keytab
--principal=kudu/127.25.109.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.60:37807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.109.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:33.031877 16271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:33.032966 16271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:33.041291 16279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:33.041522 16271 server_base.cc:1034] running on GCE node
W20260924 16:06:33.041291 16276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:33.041299 16277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:33.042140 16271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:33.043167 16271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:33.044373 16271 hybrid_clock.cc:648] HybridClock initialized: now 1790265993044353 us; error 28 us; skew 500 ppm
Sep 24 16:06:33 dist-test-slave-n106 krb5kdc[16105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265993, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:33.047055 16271 init.cc:377] Logged in from keytab as kudu/127.25.109.60@KRBTEST.COM (short username kudu)
I20260924 16:06:33.047909 16271 webserver.cc:460] Webserver started at http://127.25.109.60:43175/ using document root <none> and password file <none>
I20260924 16:06:33.048532 16271 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:33.048624 16271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:33.048830 16271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:33.050834 16271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-2/data/instance:
uuid: "9239268bf83b4730b54cac9a2ea76e6c"
format_stamp: "Formatted at 2026-09-24 16:06:33 on dist-test-slave-n106"
server_key: "2485fcdb288f5ba8e4af9326637ff4d1"
server_key_iv: "e3eda9264e37549b9ba6ce730fc7a83e"
server_key_version: "encryptionkey@0"
I20260924 16:06:33.051465 16271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-2/wal/instance:
uuid: "9239268bf83b4730b54cac9a2ea76e6c"
format_stamp: "Formatted at 2026-09-24 16:06:33 on dist-test-slave-n106"
server_key: "2485fcdb288f5ba8e4af9326637ff4d1"
server_key_iv: "e3eda9264e37549b9ba6ce730fc7a83e"
server_key_version: "encryptionkey@0"
I20260924 16:06:33.055161 16271 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 16:06:33.057564 16286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:33.058741 16271 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:06:33.058881 16271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-2/wal
uuid: "9239268bf83b4730b54cac9a2ea76e6c"
format_stamp: "Formatted at 2026-09-24 16:06:33 on dist-test-slave-n106"
server_key: "2485fcdb288f5ba8e4af9326637ff4d1"
server_key_iv: "e3eda9264e37549b9ba6ce730fc7a83e"
server_key_version: "encryptionkey@0"
I20260924 16:06:33.058988 16271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:33.086293 16271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:33.089263 16271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:33.089468 16271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:33.096822 16271 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.60:37807
I20260924 16:06:33.097134 16339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.60:37807 every 8 connection(s)
I20260924 16:06:33.097847 16271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-2/data/info.pb
I20260924 16:06:33.100715 16340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:33.102187 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 16271
I20260924 16:06:33.102317 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265708917108-26036-0/minicluster-data/master-2/wal/instance
I20260924 16:06:33.102610 26036 external_mini_cluster.cc:1428] Setting key 0eafd6f102a57182ce85b90c4955defb
I20260924 16:06:33.105268 16340 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32857 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:06:33 dist-test-slave-n106 krb5kdc[16105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.60@KRBTEST.COM
Sep 24 16:06:33 dist-test-slave-n106 krb5kdc[16105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265993, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.60@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:06:33.115489 16187 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.109.60" port: 37807 } attempt: 3
I20260924 16:06:33.116997 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.107009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42934 (local address 127.25.109.62:32857)
0924 16:06:33.107158 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.107162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.107456 (+ 294us) server_negotiation.cc:407] Connection header received
0924 16:06:33.107583 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.107586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.107635 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.107732 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:33.108381 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.108921 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.109964 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.110172 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.110287 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.113091 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:33.113094 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:33.113096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:33.113119 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:33.114701 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:33.115293 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:33.115296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:33.115298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:33.115379 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:33.115819 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:33.115825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:33.115830 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:33.116073 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:33.116169 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:33.116676 (+ 507us) server_negotiation.cc:299] Negotiation successful
0924 16:06:33.116806 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:06:33.117902 16344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.106950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:32857 (local address 127.0.0.1:42934)
0924 16:06:33.107310 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:33.107358 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:33.107486 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:33.107786 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:33.107790 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:33.108110 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:33.108227 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:33.108233 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.109054 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.109058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:33.109831 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:33.109841 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.110328 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.110332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:33.110436 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.110946 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:33.110962 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:33.112920 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:33.114839 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:33.114849 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:33.114860 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:33.115198 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:33.115518 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:33.115521 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:33.115523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:33.115672 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:33.116168 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:33.116175 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:33.116338 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 16:06:33.116598 (+ 260us) client_negotiation.cc:240] Negotiation successful
0924 16:06:33.116931 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":142,"threads_started":1}
I20260924 16:06:33.117900 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.104047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40612 (local address 127.25.109.60:37807)
0924 16:06:33.104543 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.104560 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.104597 (+ 37us) server_negotiation.cc:407] Connection header received
0924 16:06:33.105241 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.105258 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.105586 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.105979 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:33.106878 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.108188 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.109165 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.109662 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.109784 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.112588 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:33.112595 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:33.112609 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:33.112635 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:33.114681 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:33.115277 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:33.115283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:33.115285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:33.115425 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:33.115737 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:33.115741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:33.115743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:33.116257 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:33.116503 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:33.117209 (+ 706us) server_negotiation.cc:299] Negotiation successful
0924 16:06:33.117352 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":138,"threads_started":1}
I20260924 16:06:33.119203 16263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.109.60" port: 37807 } attempt: 3
I20260924 16:06:33.119560 16340 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.61" port: 37063 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:06:33 dist-test-slave-n106 krb5kdc[16105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265992, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for kudu/127.25.109.60@KRBTEST.COM
Sep 24 16:06:33 dist-test-slave-n106 krb5kdc[16105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265992, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.61@KRBTEST.COM for kudu/127.25.109.60@KRBTEST.COM
Sep 24 16:06:33 dist-test-slave-n106 krb5kdc[16105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265993, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.60@KRBTEST.COM for kudu/127.25.109.61@KRBTEST.COM
I20260924 16:06:33.127394 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.115973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:37807 (local address 127.0.0.1:40614)
0924 16:06:33.116109 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:33.116125 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:33.116205 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:33.116820 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:33.116823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:33.116993 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:33.117091 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:33.117097 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.118128 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.118132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:33.118904 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:33.118915 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.121106 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.121110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:33.121198 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.121735 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:33.121750 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:33.123303 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:33.125573 (+ 2270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:33.125576 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:33.125578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:33.125850 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:33.126241 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:33.126245 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:33.126247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:33.126348 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:33.126847 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:33.126850 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:33.126957 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 16:06:33.127059 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 16:06:33.127195 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 16:06:33.127985 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.116252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40614 (local address 127.25.109.60:37807)
0924 16:06:33.116598 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.116608 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.116620 (+ 12us) server_negotiation.cc:407] Connection header received
0924 16:06:33.116651 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.116653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.116691 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.117201 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:33.117284 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.117991 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.120647 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.120977 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.121397 (+ 420us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.123471 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:33.123474 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:33.123477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:33.123511 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:33.125462 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:33.125980 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:33.125983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:33.125984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:33.126122 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:33.126485 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:33.126488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:33.126490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:33.126721 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:33.126825 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:33.127618 (+ 793us) server_negotiation.cc:299] Negotiation successful
0924 16:06:33.127732 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":270,"spinlock_wait_cycles":3840,"thread_start_us":94,"threads_started":1}
Sep 24 16:06:33 dist-test-slave-n106 krb5kdc[16105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:06:33.130265 16344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.119983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:37063 (local address 127.0.0.1:54110)
0924 16:06:33.120155 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:33.120168 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:33.120296 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:33.120913 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:33.120919 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:33.121093 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:33.121654 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:33.121669 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.122723 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.122727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:33.123605 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:33.123618 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.124367 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.124371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:33.124504 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.124953 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:33.124967 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:33.126546 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:33.128604 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:33.128609 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:33.128611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:33.128884 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:33.129207 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:33.129209 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:33.129211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:33.129278 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:33.129773 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:33.129777 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:33.129872 (+ 95us) client_negotiation.cc:769] Sending connection context
0924 16:06:33.129988 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 16:06:33.130115 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"spinlock_wait_cycles":3328}
I20260924 16:06:33.130379 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.120065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54110 (local address 127.25.109.61:37063)
0924 16:06:33.120159 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.120163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.120224 (+ 61us) server_negotiation.cc:407] Connection header received
0924 16:06:33.120377 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.120380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.120449 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.120525 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:33.121806 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.122564 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.123744 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.124139 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.124212 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.126707 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:33.126711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:33.126714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:33.126737 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:33.128516 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:33.129008 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:33.129011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:33.129014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:33.129111 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:33.129374 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:33.129378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:33.129379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:33.129656 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:33.129772 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:33.130043 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 16:06:33.130179 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":41,"spinlock_wait_cycles":896}
I20260924 16:06:33.131487 16340 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.60" port: 37807 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:06:33.132295 16187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2b6687955c34b4083cdc1c9e2de46d6: Bootstrap starting.
I20260924 16:06:33.133685 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.121272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42938 (local address 127.25.109.62:32857)
0924 16:06:33.121753 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.121759 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.122031 (+ 272us) server_negotiation.cc:407] Connection header received
0924 16:06:33.122068 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.122071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.122109 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.122173 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:33.122993 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.124266 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.126293 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.126899 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.126970 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.130140 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:33.130143 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:33.130146 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:33.130170 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:33.131750 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:33.132250 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:33.132254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:33.132255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:33.132341 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:33.132691 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:33.132694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:33.132696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:33.132895 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:33.133060 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:33.133415 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 16:06:33.133526 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 16:06:33.133764 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.119686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:37807 (local address 127.0.0.1:40624)
0924 16:06:33.119826 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:33.119839 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:33.119945 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:33.120282 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:33.120284 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:33.120519 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:33.120630 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:33.120638 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.121616 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.121619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:33.122482 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:33.122494 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.123138 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.123142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:33.123236 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.123718 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:33.123739 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:33.125321 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:33.128866 (+ 3545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:33.128869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:33.128871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:33.129145 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:33.132744 (+ 3599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:33.132748 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:33.132750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:33.132844 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:33.133303 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:33.133307 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:33.133401 (+ 94us) client_negotiation.cc:769] Sending connection context
0924 16:06:33.133478 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 16:06:33.133608 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":4}
I20260924 16:06:33.134346 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.119708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40624 (local address 127.25.109.60:37807)
0924 16:06:33.120093 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.120097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.120109 (+ 12us) server_negotiation.cc:407] Connection header received
0924 16:06:33.120142 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.120145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.120187 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.120262 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:33.120742 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.121428 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.122635 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.123005 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.123458 (+ 453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.125473 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:33.125478 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:33.125482 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:33.125523 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:33.128110 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:33.132549 (+ 4439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:33.132552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:33.132554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:33.132634 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:33.132975 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:33.132979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:33.132982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:33.133189 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:33.134017 (+ 828us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:33.134089 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 16:06:33.134194 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":314,"spinlock_wait_cycles":384}
I20260924 16:06:33.135416 16187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2b6687955c34b4083cdc1c9e2de46d6: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:33.136492 16187 log.cc:824] T 00000000000000000000000000000000 P d2b6687955c34b4083cdc1c9e2de46d6: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:06:33 dist-test-slave-n106 krb5kdc[16105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265993, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.60@KRBTEST.COM for kudu/127.25.109.60@KRBTEST.COM
I20260924 16:06:33.138018 16263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55bc470e70254d7dbfbee77e12012007: Bootstrap starting.
I20260924 16:06:33.138823 16187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2b6687955c34b4083cdc1c9e2de46d6: No bootstrap required, opened a new log
I20260924 16:06:33.141050 16263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55bc470e70254d7dbfbee77e12012007: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:33.141127 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.132029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40636 (local address 127.25.109.60:37807)
0924 16:06:33.132156 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.132159 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.132265 (+ 106us) server_negotiation.cc:407] Connection header received
0924 16:06:33.132373 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.132377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.132457 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.132529 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:33.132990 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.133752 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.134832 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.135154 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.135246 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.137571 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:33.137574 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:33.137578 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:33.137602 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:33.139241 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:33.139720 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:33.139723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:33.139725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:33.139796 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:33.140104 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:33.140107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:33.140108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:33.140304 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:33.140450 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:33.140775 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 16:06:33.140891 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":66}
I20260924 16:06:33.141639 16344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.132004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:37807 (local address 127.0.0.1:40636)
0924 16:06:33.132198 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:33.132207 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:33.132304 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:33.132626 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:33.132629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:33.132796 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:33.132883 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:33.132892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.133869 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.133873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:33.134717 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:33.134727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.135355 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.135358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:33.135446 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.135950 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:33.135963 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:33.137437 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:33.139369 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:33.139371 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:33.139373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:33.139619 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:33.139933 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:33.139935 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:33.139937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:33.140020 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:33.140487 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:33.140490 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:33.140574 (+ 84us) client_negotiation.cc:769] Sending connection context
0924 16:06:33.141267 (+ 693us) client_negotiation.cc:240] Negotiation successful
0924 16:06:33.141399 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
Sep 24 16:06:33 dist-test-slave-n106 krb5kdc[16105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.61@KRBTEST.COM
I20260924 16:06:33.142938 16263 log.cc:824] T 00000000000000000000000000000000 P 55bc470e70254d7dbfbee77e12012007: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:33.143340 16187 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2b6687955c34b4083cdc1c9e2de46d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b6687955c34b4083cdc1c9e2de46d6" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32857 } } peers { permanent_uuid: "55bc470e70254d7dbfbee77e12012007" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 37063 } } peers { permanent_uuid: "9239268bf83b4730b54cac9a2ea76e6c" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 37807 } }
I20260924 16:06:33.143628 16187 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2b6687955c34b4083cdc1c9e2de46d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2b6687955c34b4083cdc1c9e2de46d6, State: Initialized, Role: FOLLOWER
I20260924 16:06:33.143994 16187 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2b6687955c34b4083cdc1c9e2de46d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b6687955c34b4083cdc1c9e2de46d6" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32857 } } peers { permanent_uuid: "55bc470e70254d7dbfbee77e12012007" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 37063 } } peers { permanent_uuid: "9239268bf83b4730b54cac9a2ea76e6c" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 37807 } }
I20260924 16:06:33.144675 16348 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2b6687955c34b4083cdc1c9e2de46d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b6687955c34b4083cdc1c9e2de46d6" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32857 } } peers { permanent_uuid: "55bc470e70254d7dbfbee77e12012007" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 37063 } } peers { permanent_uuid: "9239268bf83b4730b54cac9a2ea76e6c" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 37807 } } }
I20260924 16:06:33.144798 16348 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2b6687955c34b4083cdc1c9e2de46d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:06:33.145095 16340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9239268bf83b4730b54cac9a2ea76e6c: Bootstrap starting.
I20260924 16:06:33.145534 16187 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2b6687955c34b4083cdc1c9e2de46d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:33.146175 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.135725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54124 (local address 127.25.109.61:37063)
0924 16:06:33.135851 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.135854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.135869 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:06:33.135989 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.135992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.136046 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.136095 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:33.136983 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.137730 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.138746 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.139065 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.139150 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.142297 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:33.142301 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:33.142304 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:33.142329 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:33.144209 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:33.144744 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:33.144747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:33.144749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:33.144839 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:33.145177 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:33.145181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:33.145182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:33.145388 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:33.145561 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:33.145911 (+ 350us) server_negotiation.cc:299] Negotiation successful
0924 16:06:33.146026 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 16:06:33.146739 16263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55bc470e70254d7dbfbee77e12012007: No bootstrap required, opened a new log
W20260924 16:06:33.147405 16139 tablet.cc:2369] T 00000000000000000000000000000000 P d2b6687955c34b4083cdc1c9e2de46d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:06:33.148741 16340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9239268bf83b4730b54cac9a2ea76e6c: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:33.149674 16340 log.cc:824] T 00000000000000000000000000000000 P 9239268bf83b4730b54cac9a2ea76e6c: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:33.150880 16263 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55bc470e70254d7dbfbee77e12012007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b6687955c34b4083cdc1c9e2de46d6" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32857 } } peers { permanent_uuid: "55bc470e70254d7dbfbee77e12012007" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 37063 } } peers { permanent_uuid: "9239268bf83b4730b54cac9a2ea76e6c" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 37807 } }
I20260924 16:06:33.151105 16263 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55bc470e70254d7dbfbee77e12012007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55bc470e70254d7dbfbee77e12012007, State: Initialized, Role: FOLLOWER
I20260924 16:06:33.151436 26036 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:06:33.151436 16263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55bc470e70254d7dbfbee77e12012007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b6687955c34b4083cdc1c9e2de46d6" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32857 } } peers { permanent_uuid: "55bc470e70254d7dbfbee77e12012007" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 37063 } } peers { permanent_uuid: "9239268bf83b4730b54cac9a2ea76e6c" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 37807 } }
W20260924 16:06:33.151657 16359 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d2b6687955c34b4083cdc1c9e2de46d6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:06:33.151726 16359 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:06:33.152706 16160 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d2b6687955c34b4083cdc1c9e2de46d6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42938
I20260924 16:06:33.152770 16263 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55bc470e70254d7dbfbee77e12012007 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:33.152894 16160 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2b6687955c34b4083cdc1c9e2de46d6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 16:06:33.152993 16160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2b6687955c34b4083cdc1c9e2de46d6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:33.154345 16160 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2b6687955c34b4083cdc1c9e2de46d6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b6687955c34b4083cdc1c9e2de46d6" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32857 } } peers { permanent_uuid: "55bc470e70254d7dbfbee77e12012007" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 37063 } } peers { permanent_uuid: "9239268bf83b4730b54cac9a2ea76e6c" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 37807 } }
I20260924 16:06:33.155232 16160 leader_election.cc:288] T 00000000000000000000000000000000 P d2b6687955c34b4083cdc1c9e2de46d6 [CANDIDATE]: Term 1 election: Requested vote from peers 55bc470e70254d7dbfbee77e12012007 (127.25.109.61:37063), 9239268bf83b4730b54cac9a2ea76e6c (127.25.109.60:37807)
I20260924 16:06:33.156740 16362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55bc470e70254d7dbfbee77e12012007 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b6687955c34b4083cdc1c9e2de46d6" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32857 } } peers { permanent_uuid: "55bc470e70254d7dbfbee77e12012007" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 37063 } } peers { permanent_uuid: "9239268bf83b4730b54cac9a2ea76e6c" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 37807 } } }
I20260924 16:06:33.156924 16362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55bc470e70254d7dbfbee77e12012007 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:06:33.156767 16237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d2b6687955c34b4083cdc1c9e2de46d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "55bc470e70254d7dbfbee77e12012007"
I20260924 16:06:33.157084 16237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55bc470e70254d7dbfbee77e12012007 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:33.158269 16237 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 55bc470e70254d7dbfbee77e12012007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2b6687955c34b4083cdc1c9e2de46d6 in term 1.
I20260924 16:06:33.158730 16135 leader_election.cc:302] T 00000000000000000000000000000000 P d2b6687955c34b4083cdc1c9e2de46d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55bc470e70254d7dbfbee77e12012007, d2b6687955c34b4083cdc1c9e2de46d6; no voters:
I20260924 16:06:33.159024 16348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2b6687955c34b4083cdc1c9e2de46d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:33.159248 16348 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2b6687955c34b4083cdc1c9e2de46d6 [term 1 LEADER]: Becoming Leader. State: Replica: d2b6687955c34b4083cdc1c9e2de46d6, State: Running, Role: LEADER
I20260924 16:06:33.159595 16348 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2b6687955c34b4083cdc1c9e2de46d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b6687955c34b4083cdc1c9e2de46d6" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32857 } } peers { permanent_uuid: "55bc470e70254d7dbfbee77e12012007" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 37063 } } peers { permanent_uuid: "9239268bf83b4730b54cac9a2ea76e6c" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 37807 } }
I20260924 16:06:33.152616 16118 master_runner.cc:186] Error getting master registration for 127.25.109.61:37063: OK, instance_id { permanent_uuid: "55bc470e70254d7dbfbee77e12012007" instance_seqno: 1790265992863189 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 16:06:33.160919 16118 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 16:06:33.161526 16378 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2b6687955c34b4083cdc1c9e2de46d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2b6687955c34b4083cdc1c9e2de46d6. Latest consensus state: current_term: 1 leader_uuid: "d2b6687955c34b4083cdc1c9e2de46d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b6687955c34b4083cdc1c9e2de46d6" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32857 } } peers { permanent_uuid: "55bc470e70254d7dbfbee77e12012007" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 37063 } } peers { permanent_uuid: "9239268bf83b4730b54cac9a2ea76e6c" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 37807 } } }
I20260924 16:06:33.161737 16378 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2b6687955c34b4083cdc1c9e2de46d6 [sys.catalog]: This master's current role is: LEADER
W20260924 16:06:33.161976 16373 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 55bc470e70254d7dbfbee77e12012007: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:06:33.162060 16373 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:06:33.165921 16381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:33.168020 16340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9239268bf83b4730b54cac9a2ea76e6c: No bootstrap required, opened a new log
I20260924 16:06:33.166525 16194 master_runner.cc:186] Error getting master registration for 127.25.109.60:37807: OK, instance_id { permanent_uuid: "9239268bf83b4730b54cac9a2ea76e6c" instance_seqno: 1790265993094831 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 16:06:33.169883 16194 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 16:06:33.172304 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.162423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42942 (local address 127.25.109.62:32857)
0924 16:06:33.162560 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.162565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.162747 (+ 182us) server_negotiation.cc:407] Connection header received
0924 16:06:33.162920 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.162924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.162976 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.163033 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:33.164191 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.164954 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.166215 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.166551 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.166628 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.168633 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:33.168637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:33.168640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:33.168665 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:33.170350 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:33.170902 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:33.170906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:33.170908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:33.171006 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:33.171320 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:33.171323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:33.171325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:33.171556 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:33.171689 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:33.172008 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 16:06:33.172142 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 16:06:33.172768 16340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9239268bf83b4730b54cac9a2ea76e6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b6687955c34b4083cdc1c9e2de46d6" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32857 } } peers { permanent_uuid: "55bc470e70254d7dbfbee77e12012007" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 37063 } } peers { permanent_uuid: "9239268bf83b4730b54cac9a2ea76e6c" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 37807 } }
I20260924 16:06:33.173069 16340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9239268bf83b4730b54cac9a2ea76e6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9239268bf83b4730b54cac9a2ea76e6c, State: Initialized, Role: FOLLOWER
I20260924 16:06:33.173431 16340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9239268bf83b4730b54cac9a2ea76e6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b6687955c34b4083cdc1c9e2de46d6" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32857 } } peers { permanent_uuid: "55bc470e70254d7dbfbee77e12012007" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 37063 } } peers { permanent_uuid: "9239268bf83b4730b54cac9a2ea76e6c" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 37807 } }
I20260924 16:06:33.174443 16385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9239268bf83b4730b54cac9a2ea76e6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b6687955c34b4083cdc1c9e2de46d6" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32857 } } peers { permanent_uuid: "55bc470e70254d7dbfbee77e12012007" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 37063 } } peers { permanent_uuid: "9239268bf83b4730b54cac9a2ea76e6c" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 37807 } } }
I20260924 16:06:33.175050 16340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9239268bf83b4730b54cac9a2ea76e6c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:33.175285 16314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d2b6687955c34b4083cdc1c9e2de46d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9239268bf83b4730b54cac9a2ea76e6c"
I20260924 16:06:33.175527 16314 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9239268bf83b4730b54cac9a2ea76e6c [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:33.176950 16314 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9239268bf83b4730b54cac9a2ea76e6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2b6687955c34b4083cdc1c9e2de46d6 in term 1.
I20260924 16:06:33.177469 16381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:33.175096 16385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9239268bf83b4730b54cac9a2ea76e6c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:06:33.182955 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.162860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54132 (local address 127.25.109.61:37063)
0924 16:06:33.162966 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.162970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.172329 (+ 9359us) server_negotiation.cc:407] Connection header received
0924 16:06:33.173333 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.173337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.173388 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.173471 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:33.174272 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.175060 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.176215 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.176585 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.176669 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.179073 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:33.179078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:33.179081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:33.179112 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:33.180871 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:33.181487 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:33.181491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:33.181493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:33.181586 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:33.181913 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:33.181917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:33.181919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:33.182159 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:33.182287 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:33.182653 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 16:06:33.182778 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 16:06:33.183633 16314 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9239268bf83b4730b54cac9a2ea76e6c [term 1 FOLLOWER]: Refusing update from remote peer d2b6687955c34b4083cdc1c9e2de46d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:06:33.184263 16348 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d2b6687955c34b4083cdc1c9e2de46d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9239268bf83b4730b54cac9a2ea76e6c" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:06:33.189175 16385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9239268bf83b4730b54cac9a2ea76e6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2b6687955c34b4083cdc1c9e2de46d6. Latest consensus state: current_term: 1 leader_uuid: "d2b6687955c34b4083cdc1c9e2de46d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b6687955c34b4083cdc1c9e2de46d6" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32857 } } peers { permanent_uuid: "55bc470e70254d7dbfbee77e12012007" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 37063 } } peers { permanent_uuid: "9239268bf83b4730b54cac9a2ea76e6c" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 37807 } } }
I20260924 16:06:33.189312 16385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9239268bf83b4730b54cac9a2ea76e6c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:06:33.189594 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.163832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40648 (local address 127.25.109.60:37807)
0924 16:06:33.163943 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.163947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.180459 (+ 16512us) server_negotiation.cc:407] Connection header received
0924 16:06:33.180647 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.180650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.180700 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.180748 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:33.181628 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.182313 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.183823 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.184144 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.184222 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.186034 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:33.186039 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:33.186042 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:33.186070 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:33.187710 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:33.188202 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:33.188205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:33.188207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:33.188290 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:33.188629 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:33.188633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:33.188635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:33.188855 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:33.188965 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:33.189320 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 16:06:33.189439 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 16:06:33.190301 16237 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 55bc470e70254d7dbfbee77e12012007 [term 1 FOLLOWER]: Refusing update from remote peer d2b6687955c34b4083cdc1c9e2de46d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:06:33.191845 16348 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d2b6687955c34b4083cdc1c9e2de46d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55bc470e70254d7dbfbee77e12012007" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:06:33.192987 16348 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2b6687955c34b4083cdc1c9e2de46d6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d2b6687955c34b4083cdc1c9e2de46d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b6687955c34b4083cdc1c9e2de46d6" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32857 } } peers { permanent_uuid: "55bc470e70254d7dbfbee77e12012007" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 37063 } } peers { permanent_uuid: "9239268bf83b4730b54cac9a2ea76e6c" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 37807 } } }
I20260924 16:06:33.193106 16348 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2b6687955c34b4083cdc1c9e2de46d6 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:33.193364 16391 mvcc.cc:205] Tried to move back new op lower bound from 7332929508072296448 to 7332929507985272832. Current Snapshot: MvccSnapshot[applied={T|T < 7332929508072296448}]
I20260924 16:06:33.193377 16385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9239268bf83b4730b54cac9a2ea76e6c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d2b6687955c34b4083cdc1c9e2de46d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b6687955c34b4083cdc1c9e2de46d6" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32857 } } peers { permanent_uuid: "55bc470e70254d7dbfbee77e12012007" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 37063 } } peers { permanent_uuid: "9239268bf83b4730b54cac9a2ea76e6c" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 37807 } } }
I20260924 16:06:33.193480 16385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9239268bf83b4730b54cac9a2ea76e6c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:06:33.196874 16362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55bc470e70254d7dbfbee77e12012007 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2b6687955c34b4083cdc1c9e2de46d6. Latest consensus state: current_term: 1 leader_uuid: "d2b6687955c34b4083cdc1c9e2de46d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b6687955c34b4083cdc1c9e2de46d6" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32857 } } peers { permanent_uuid: "55bc470e70254d7dbfbee77e12012007" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 37063 } } peers { permanent_uuid: "9239268bf83b4730b54cac9a2ea76e6c" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 37807 } } }
I20260924 16:06:33.196986 16397 mvcc.cc:205] Tried to move back new op lower bound from 7332929508072296448 to 7332929507985272832. Current Snapshot: MvccSnapshot[applied={T|T < 7332929508072296448}]
I20260924 16:06:33.197037 16362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55bc470e70254d7dbfbee77e12012007 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:06:33.199033 16378 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2b6687955c34b4083cdc1c9e2de46d6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d2b6687955c34b4083cdc1c9e2de46d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b6687955c34b4083cdc1c9e2de46d6" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32857 } } peers { permanent_uuid: "55bc470e70254d7dbfbee77e12012007" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 37063 } } peers { permanent_uuid: "9239268bf83b4730b54cac9a2ea76e6c" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 37807 } } }
I20260924 16:06:33.199177 16378 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2b6687955c34b4083cdc1c9e2de46d6 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:33.200276 16362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55bc470e70254d7dbfbee77e12012007 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d2b6687955c34b4083cdc1c9e2de46d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b6687955c34b4083cdc1c9e2de46d6" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 32857 } } peers { permanent_uuid: "55bc470e70254d7dbfbee77e12012007" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 37063 } } peers { permanent_uuid: "9239268bf83b4730b54cac9a2ea76e6c" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 37807 } } }
I20260924 16:06:33.200387 16362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55bc470e70254d7dbfbee77e12012007 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 16:06:33.202170 16399 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9239268bf83b4730b54cac9a2ea76e6c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:06:33.202250 16399 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:06:33.203899 16381 catalog_manager.cc:1347] Generated new cluster ID: c886dd0b5a174effa24d480747917d6d
I20260924 16:06:33.203960 16381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:33.218263 16381 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:33.220562 16381 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:33.236059 16381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2b6687955c34b4083cdc1c9e2de46d6: Generated new TSK 0
I20260924 16:06:33.237077 16381 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 16:06:33.263815 16343 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:33.260250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40654 (local address 127.25.109.60:37807)
0924 16:06:33.260395 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.260400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.262591 (+ 2191us) server_negotiation.cc:407] Connection header received
0924 16:06:33.262811 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.262814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.262868 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.262935 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:33.263664 (+ 729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40654: BlockingRecv error: recv got EOF from 127.0.0.1:40654 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 16:06:33.265287 16269 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:33.261152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54134 (local address 127.25.109.61:37063)
0924 16:06:33.261271 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.261275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.264314 (+ 3039us) server_negotiation.cc:407] Connection header received
0924 16:06:33.264542 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.264545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.264599 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.264649 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:33.265159 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54134: BlockingRecv error: recv got EOF from 127.0.0.1:54134 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260924 16:06:33.267058 16190 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:33.261878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42944 (local address 127.25.109.62:32857)
0924 16:06:33.262013 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.262018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.265667 (+ 3649us) server_negotiation.cc:407] Connection header received
0924 16:06:33.266055 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.266059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.266114 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.266344 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:33.266935 (+ 591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42944: BlockingRecv error: recv got EOF from 127.0.0.1:42944 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260924 16:06:33.271783 16269 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:33.270200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54136 (local address 127.25.109.61:37063)
0924 16:06:33.270305 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.270309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.270748 (+ 439us) server_negotiation.cc:407] Connection header received
0924 16:06:33.270990 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.270993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.271072 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.271126 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:33.271625 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54136: BlockingRecv error: recv got EOF from 127.0.0.1:54136 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 16:06:33.277070 16269 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:33.275483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54150 (local address 127.25.109.61:37063)
0924 16:06:33.275588 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.275592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.276114 (+ 522us) server_negotiation.cc:407] Connection header received
0924 16:06:33.276303 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.276307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.276357 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.276438 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:33.276960 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54150: BlockingRecv error: recv got EOF from 127.0.0.1:54150 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 16:06:33.282881 16269 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:33.281370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54164 (local address 127.25.109.61:37063)
0924 16:06:33.281469 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.281473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.281978 (+ 505us) server_negotiation.cc:407] Connection header received
0924 16:06:33.282170 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.282173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.282222 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.282278 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:33.282763 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54164: BlockingRecv error: recv got EOF from 127.0.0.1:54164 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260924 16:06:33.291098 16269 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:33.289353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54178 (local address 127.25.109.61:37063)
0924 16:06:33.289518 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.289523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.290105 (+ 582us) server_negotiation.cc:407] Connection header received
0924 16:06:33.290315 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.290319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.290371 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.290424 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:33.290975 (+ 551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54178: BlockingRecv error: recv got EOF from 127.0.0.1:54178 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260924 16:06:33.303582 16269 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:33.301830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54186 (local address 127.25.109.61:37063)
0924 16:06:33.301960 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.301965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.302536 (+ 571us) server_negotiation.cc:407] Connection header received
0924 16:06:33.302750 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.302754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.302814 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.302869 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:33.303452 (+ 583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54186: BlockingRecv error: recv got EOF from 127.0.0.1:54186 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 16:06:33.324278 16269 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:33.322434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54188 (local address 127.25.109.61:37063)
0924 16:06:33.322569 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.322574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.323201 (+ 627us) server_negotiation.cc:407] Connection header received
0924 16:06:33.323444 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.323448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.323513 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.323567 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:33.324157 (+ 590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54188: BlockingRecv error: recv got EOF from 127.0.0.1:54188 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 16:06:33.360986 16269 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:33.359185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54204 (local address 127.25.109.61:37063)
0924 16:06:33.359345 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.359350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.359974 (+ 624us) server_negotiation.cc:407] Connection header received
0924 16:06:33.360183 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.360187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.360240 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.360290 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:33.360854 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54204: BlockingRecv error: recv got EOF from 127.0.0.1:54204 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260924 16:06:33.428939 16269 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:33.427621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54218 (local address 127.25.109.61:37063)
0924 16:06:33.427801 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.427805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.427955 (+ 150us) server_negotiation.cc:407] Connection header received
0924 16:06:33.428150 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.428154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.428213 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.428309 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:33.428812 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54218: BlockingRecv error: recv got EOF from 127.0.0.1:54218 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 16:06:33.560195 16269 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:33.559053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54222 (local address 127.25.109.61:37063)
0924 16:06:33.559207 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.559213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.559390 (+ 177us) server_negotiation.cc:407] Connection header received
0924 16:06:33.559505 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.559508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.559556 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.559628 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:33.560054 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54222: BlockingRecv error: recv got EOF from 127.0.0.1:54222 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 16:06:33.819547 16269 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:33.818496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54232 (local address 127.25.109.61:37063)
0924 16:06:33.818631 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.818634 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.818730 (+ 96us) server_negotiation.cc:407] Connection header received
0924 16:06:33.818897 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.818901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.818955 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.819042 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:33.819447 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54232: BlockingRecv error: recv got EOF from 127.0.0.1:54232 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 16:06:34.163501 16373 catalog_manager.cc:1259] Loaded cluster ID: c886dd0b5a174effa24d480747917d6d
I20260924 16:06:34.163585 16373 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 55bc470e70254d7dbfbee77e12012007: loading cluster ID for follower catalog manager: success
I20260924 16:06:34.165381 16373 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 55bc470e70254d7dbfbee77e12012007: acquiring CA information for follower catalog manager: success
I20260924 16:06:34.166527 16373 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 55bc470e70254d7dbfbee77e12012007: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:06:34.203472 16399 catalog_manager.cc:1259] Loaded cluster ID: c886dd0b5a174effa24d480747917d6d
I20260924 16:06:34.203552 16399 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9239268bf83b4730b54cac9a2ea76e6c: loading cluster ID for follower catalog manager: success
I20260924 16:06:34.205259 16399 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9239268bf83b4730b54cac9a2ea76e6c: acquiring CA information for follower catalog manager: success
I20260924 16:06:34.206419 16399 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9239268bf83b4730b54cac9a2ea76e6c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:06:34.337530 16474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:34.333747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56020 (local address 127.25.109.61:37063)
0924 16:06:34.334003 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:34.334007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:34.334024 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:06:34.334198 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:34.334202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:34.334337 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:34.334448 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:06:34.334684 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:34.335277 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:34.336109 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:34.336328 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:34.336469 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:34.336752 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:06:34.337204 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:06:34.337298 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:34.337352 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 16:06:34.337403 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":100,"threads_started":1}
I20260924 16:06:34.345304 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:34.340584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44528 (local address 127.25.109.60:37807)
0924 16:06:34.340829 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:34.340833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:34.340927 (+ 94us) server_negotiation.cc:407] Connection header received
0924 16:06:34.341065 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:34.341068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:34.341234 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:34.341384 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:06:34.341705 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:34.342363 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:34.343529 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:34.343889 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:34.344005 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:34.344354 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:06:34.344878 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:06:34.345028 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:34.345073 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 16:06:34.345133 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":64,"threads_started":1}
I20260924 16:06:34.352880 16474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:34.348376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56032 (local address 127.25.109.61:37063)
0924 16:06:34.348540 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:34.348544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:34.348921 (+ 377us) server_negotiation.cc:407] Connection header received
0924 16:06:34.349119 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:34.349122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:34.349176 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:34.349252 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:06:34.349817 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:34.350580 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:34.351637 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:34.351995 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:34.352099 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:34.352361 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:06:34.352477 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:06:34.352543 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:34.352699 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 16:06:34.352757 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 16:06:34.352929 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:34.348342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44536 (local address 127.25.109.60:37807)
0924 16:06:34.348465 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:34.348470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:34.348910 (+ 440us) server_negotiation.cc:407] Connection header received
0924 16:06:34.349134 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:34.349137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:34.349183 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:34.349252 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:06:34.349765 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:34.350544 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:34.351681 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:34.351982 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:34.352055 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:34.352415 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:06:34.352516 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:06:34.352561 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:34.352747 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 16:06:34.352808 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 16:06:34.352941 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:34.348255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50458 (local address 127.25.109.62:32857)
0924 16:06:34.348938 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:34.348942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:34.348959 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:06:34.349056 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:34.349059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:34.349265 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:34.349385 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:06:34.349587 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:34.350207 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:34.351257 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:34.351622 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:34.351723 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:34.352026 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:06:34.352428 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:06:34.352516 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:34.352603 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 16:06:34.352671 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":71,"threads_started":1}
I20260924 16:06:34.356436 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 16118
I20260924 16:06:34.363586 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 16194
I20260924 16:06:34.369812 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 16271
2026-09-24T16:06:34Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3055 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:06:34 dist-test-slave-n106 krb5kdc[16492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:06:34 dist-test-slave-n106 krb5kdc[16492](info): set up 2 sockets
Sep 24 16:06:34 dist-test-slave-n106 krb5kdc[16492](info): commencing operation
krb5kdc: starting...
W20260924 16:06:35.417811 26036 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/krb5kdc/test-user.keytab.
Sep 24 16:06:35 dist-test-slave-n106 krb5kdc[16492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:06:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
Entry for principal kudu/127.25.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab.
I20260924 16:06:35.539186 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:44327
--webserver_interface=127.25.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:34671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:44327,127.25.109.61:36745,127.25.109.60:39015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:35.648602 16505 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 16:06:35.648934 16505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:35.649010 16505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:35.649097 16505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:35.653023 16505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:35.653112 16505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:35.653151 16505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:35.653187 16505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:35.653223 16505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:35.658371 16505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:44327,127.25.109.61:36745,127.25.109.60:39015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.62.keytab
--principal=kudu/127.25.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.62:44327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:35.659427 16505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:35.660470 16505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:35.669179 16512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:35.669217 16505 server_base.cc:1034] running on GCE node
W20260924 16:06:35.669224 16511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:35.669342 16514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:35.670010 16505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:35.671072 16505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:35.672233 16505 hybrid_clock.cc:648] HybridClock initialized: now 1790265995672214 us; error 29 us; skew 500 ppm
Sep 24 16:06:35 dist-test-slave-n106 krb5kdc[16492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265995, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:35.674860 16505 init.cc:377] Logged in from keytab as kudu/127.25.109.62@KRBTEST.COM (short username kudu)
I20260924 16:06:35.675719 16505 webserver.cc:460] Webserver started at http://127.25.109.62:34269/ using document root <none> and password file <none>
I20260924 16:06:35.676270 16505 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:35.676329 16505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:35.676589 16505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:35.678612 16505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-0/data/instance:
uuid: "b4eaea955d16422a82c9fe17eb97dc54"
format_stamp: "Formatted at 2026-09-24 16:06:35 on dist-test-slave-n106"
server_key: "5b602feb14e9c759ea5321a36bc4e6f6"
server_key_iv: "973f481afc95ed9ef7b7c28399339241"
server_key_version: "encryptionkey@0"
I20260924 16:06:35.679193 16505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-0/wal/instance:
uuid: "b4eaea955d16422a82c9fe17eb97dc54"
format_stamp: "Formatted at 2026-09-24 16:06:35 on dist-test-slave-n106"
server_key: "5b602feb14e9c759ea5321a36bc4e6f6"
server_key_iv: "973f481afc95ed9ef7b7c28399339241"
server_key_version: "encryptionkey@0"
I20260924 16:06:35.682968 16505 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 16:06:35.685456 16521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:35.686681 16505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:06:35.686813 16505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-0/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-0/wal
uuid: "b4eaea955d16422a82c9fe17eb97dc54"
format_stamp: "Formatted at 2026-09-24 16:06:35 on dist-test-slave-n106"
server_key: "5b602feb14e9c759ea5321a36bc4e6f6"
server_key_iv: "973f481afc95ed9ef7b7c28399339241"
server_key_version: "encryptionkey@0"
I20260924 16:06:35.686919 16505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:35.697887 16505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:35.700850 16505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:35.701049 16505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:35.707736 16505 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.62:44327
I20260924 16:06:35.708019 16573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.62:44327 every 8 connection(s)
I20260924 16:06:35.708819 16505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:35.711519 16574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:35.715167 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 16505
I20260924 16:06:35.715306 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-0/wal/instance
I20260924 16:06:35.715631 26036 external_mini_cluster.cc:1428] Setting key 714a05c13ec3ed73c0790b8941eeccdc
I20260924 16:06:35.715361 16574 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44327 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.25.109.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.61@KRBTEST.COM" created.
Sep 24 16:06:35 dist-test-slave-n106 krb5kdc[16492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265995, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:06:35.732662 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:35.717990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:44327 (local address 127.0.0.1:46404)
0924 16:06:35.718558 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:35.718594 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:35.719410 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:35.720163 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:35.720168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:35.720642 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:35.720868 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:35.720882 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:35.722028 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:35.722031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:35.722762 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:35.722771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:35.723416 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:35.723420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:35.723531 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:35.724114 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:35.724131 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:35.727276 (+ 3145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:35.729658 (+ 2382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:35.729667 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:35.729674 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:35.729902 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:35.730318 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:35.730321 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:35.730322 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:35.730431 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:35.731231 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:35.731236 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:35.731356 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 16:06:35.731490 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 16:06:35.731757 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"spinlock_wait_cycles":5504,"thread_start_us":165,"threads_started":1}
I20260924 16:06:35.732666 16577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:35.717990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46404 (local address 127.25.109.62:44327)
0924 16:06:35.718396 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:35.718417 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:35.718705 (+ 288us) server_negotiation.cc:407] Connection header received
0924 16:06:35.719720 (+ 1015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:35.719724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:35.719977 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:35.720171 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:35.721016 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:35.721885 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:35.722887 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:35.723249 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:35.723428 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:35.727437 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:35.727443 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:35.727453 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:35.727476 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:35.729543 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:35.730026 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:35.730036 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:35.730041 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:35.730206 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:35.730534 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:35.730540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:35.730545 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:35.731101 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:35.731256 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:35.731622 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 16:06:35.731872 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":269,"thread_start_us":145,"threads_started":1}
I20260924 16:06:35.735293 16574 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.61" port: 36745 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:06:35.735996 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:35.735749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:36745 (local address 127.0.0.1:43396)
0924 16:06:35.735872 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:35.735940 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.61:36745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 16:06:35.736192 16524 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.61:36745: connect: Connection refused (error 111)
I20260924 16:06:35.737012 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:35.736827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:36745 (local address 127.0.0.1:43404)
0924 16:06:35.736922 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:35.736964 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.61:36745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 16:06:35.737233 16574 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.109.61:36745: Network error: Client connection negotiation failed: client connection to 127.25.109.61:36745: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.61.keytab.
Entry for principal kudu/127.25.109.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.61.keytab.
I20260924 16:06:35.743058 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.61:36745
--webserver_interface=127.25.109.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.61.keytab
--principal=kudu/127.25.109.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:34671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:44327,127.25.109.61:36745,127.25.109.60:39015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:06:35.786545 16574 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.109.61" port: 36745 } attempt: 1
I20260924 16:06:35.787525 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:35.787163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:36745 (local address 127.0.0.1:43418)
0924 16:06:35.787346 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:35.787425 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.61:36745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 16:06:35.788301 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:35.788070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:36745 (local address 127.0.0.1:43430)
0924 16:06:35.788180 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:35.788235 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.61:36745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 16:06:35.788483 16574 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.109.61:36745: Network error: Client connection negotiation failed: client connection to 127.25.109.61:36745: connect: Connection refused (error 111)
W20260924 16:06:35.855674 16582 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 16:06:35.856021 16582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:35.856099 16582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:35.856181 16582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 16:06:35.856659 16574 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.109.61" port: 36745 } attempt: 2
I20260924 16:06:35.857533 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:35.857196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:36745 (local address 127.0.0.1:43432)
0924 16:06:35.857369 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:35.857443 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.61:36745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 16:06:35.858286 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:35.858046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:36745 (local address 127.0.0.1:43434)
0924 16:06:35.858171 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:35.858224 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.61:36745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 16:06:35.858418 16574 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.109.61:36745: Network error: Client connection negotiation failed: client connection to 127.25.109.61:36745: connect: Connection refused (error 111)
W20260924 16:06:35.860843 16582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:35.860890 16582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:35.860932 16582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:35.860972 16582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:35.861019 16582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:35.866184 16582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:44327,127.25.109.61:36745,127.25.109.60:39015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.61.keytab
--principal=kudu/127.25.109.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.61:36745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.109.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:35.867146 16582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:35.868129 16582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:35.876459 16587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:35.876472 16590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:35.876471 16588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:35.876503 16582 server_base.cc:1034] running on GCE node
I20260924 16:06:35.877244 16582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:35.878289 16582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:35.879482 16582 hybrid_clock.cc:648] HybridClock initialized: now 1790265995879469 us; error 25 us; skew 500 ppm
Sep 24 16:06:35 dist-test-slave-n106 krb5kdc[16492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265995, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:35.882211 16582 init.cc:377] Logged in from keytab as kudu/127.25.109.61@KRBTEST.COM (short username kudu)
I20260924 16:06:35.883060 16582 webserver.cc:460] Webserver started at http://127.25.109.61:39095/ using document root <none> and password file <none>
I20260924 16:06:35.883649 16582 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:35.883742 16582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:35.883978 16582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:35.886071 16582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-1/data/instance:
uuid: "183d364a616c4b37857e40285941660d"
format_stamp: "Formatted at 2026-09-24 16:06:35 on dist-test-slave-n106"
server_key: "b93e27b4e507de9bc0981883eb73558c"
server_key_iv: "6e66cba0910a3166214a7aa50aa99cfd"
server_key_version: "encryptionkey@0"
I20260924 16:06:35.886654 16582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-1/wal/instance:
uuid: "183d364a616c4b37857e40285941660d"
format_stamp: "Formatted at 2026-09-24 16:06:35 on dist-test-slave-n106"
server_key: "b93e27b4e507de9bc0981883eb73558c"
server_key_iv: "6e66cba0910a3166214a7aa50aa99cfd"
server_key_version: "encryptionkey@0"
I20260924 16:06:35.890388 16582 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 16:06:35.893283 16597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:35.894423 16582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:06:35.894573 16582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-1/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-1/wal
uuid: "183d364a616c4b37857e40285941660d"
format_stamp: "Formatted at 2026-09-24 16:06:35 on dist-test-slave-n106"
server_key: "b93e27b4e507de9bc0981883eb73558c"
server_key_iv: "6e66cba0910a3166214a7aa50aa99cfd"
server_key_version: "encryptionkey@0"
I20260924 16:06:35.894686 16582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:35.932646 16582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:35.935519 16582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:35.935748 16582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:35.942730 16582 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.61:36745
I20260924 16:06:35.943017 16649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.61:36745 every 8 connection(s)
I20260924 16:06:35.943774 16582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-1/data/info.pb
I20260924 16:06:35.946564 16650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:35.949604 16574 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.109.61" port: 36745 } attempt: 3
I20260924 16:06:35.949692 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 16582
I20260924 16:06:35.949819 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-1/wal/instance
I20260924 16:06:35.950150 26036 external_mini_cluster.cc:1428] Setting key 93140d9ecf2df4b1eab232a9c1597fa6
I20260924 16:06:35.950546 16650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44327 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:06:35 dist-test-slave-n106 krb5kdc[16492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265995, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.61@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
Sep 24 16:06:35 dist-test-slave-n106 krb5kdc[16492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265995, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for kudu/127.25.109.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.109.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.109.60@KRBTEST.COM" created.
I20260924 16:06:35.964097 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:35.950165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:36745 (local address 127.0.0.1:43436)
0924 16:06:35.950383 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:35.950400 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:35.950497 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:35.953169 (+ 2672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:35.953173 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:35.953377 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:35.953539 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:35.953545 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:35.955820 (+ 2275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:35.955824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:35.956589 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:35.956606 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:35.957314 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:35.957321 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:35.957458 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:35.958015 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:35.958031 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:35.959857 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:35.962371 (+ 2514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:35.962374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:35.962376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:35.962658 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:35.962960 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:35.962967 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:35.962972 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:35.963075 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:35.963560 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:35.963563 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:35.963663 (+ 100us) client_negotiation.cc:769] Sending connection context
0924 16:06:35.963780 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 16:06:35.963913 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":2}
I20260924 16:06:35.964628 16577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:35.952723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46408 (local address 127.25.109.62:44327)
0924 16:06:35.952859 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:35.952865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:35.953292 (+ 427us) server_negotiation.cc:407] Connection header received
0924 16:06:35.953483 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:35.953486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:35.953534 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:35.953622 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:35.954459 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:35.954983 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:35.956287 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:35.956833 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:35.956963 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:35.959585 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:35.959591 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:35.959598 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:35.959634 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:35.961667 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:35.962212 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:35.962218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:35.962219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:35.962301 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:35.962672 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:35.962677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:35.962679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:35.962939 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:35.963028 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:35.963698 (+ 670us) server_negotiation.cc:299] Negotiation successful
0924 16:06:35.963843 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":42,"spinlock_wait_cycles":12032}
I20260924 16:06:35.965248 16652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:35.950905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43436 (local address 127.25.109.61:36745)
0924 16:06:35.951363 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:35.951390 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:35.951428 (+ 38us) server_negotiation.cc:407] Connection header received
0924 16:06:35.952329 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:35.952353 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:35.952839 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:35.953720 (+ 881us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:35.954169 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:35.955664 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:35.956712 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:35.957183 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:35.957280 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:35.960287 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:35.960294 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:35.960308 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:35.960335 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:35.962244 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:35.962772 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:35.962778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:35.962780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:35.962855 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:35.963187 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:35.963190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:35.963192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:35.963450 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:35.963591 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:35.963877 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 16:06:35.964182 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":312,"thread_start_us":179,"threads_started":1}
I20260924 16:06:35.965252 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:35.952703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:44327 (local address 127.0.0.1:46408)
0924 16:06:35.953147 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:35.953203 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:35.953385 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:35.953672 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:35.953681 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:35.954066 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:35.954321 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:35.954331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:35.955097 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:35.955108 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:35.956137 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:35.956148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:35.957003 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:35.957007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:35.957183 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:35.957858 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:35.957883 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:35.959420 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:35.961781 (+ 2361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:35.961792 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:35.961802 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:35.962105 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:35.962396 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:35.962400 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:35.962402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:35.962542 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:35.963023 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:35.963030 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:35.963377 (+ 347us) client_negotiation.cc:769] Sending connection context
0924 16:06:35.963953 (+ 576us) client_negotiation.cc:240] Negotiation successful
0924 16:06:35.964252 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":73,"spinlock_wait_cycles":768,"thread_start_us":147,"threads_started":1}
I20260924 16:06:35.966687 16650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.61" port: 36745 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:06:35.967420 16574 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.60" port: 39015 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:06:35.968206 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:35.967881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:39015 (local address 127.0.0.1:58752)
0924 16:06:35.968045 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:35.968139 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:39015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 16:06:35.968907 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:35.968664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:39015 (local address 127.0.0.1:58764)
0924 16:06:35.968797 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:35.968845 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:39015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260924 16:06:35.969031 16574 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.109.60:39015: Network error: Client connection negotiation failed: client connection to 127.25.109.60:39015: connect: Connection refused (error 111)
Sep 24 16:06:35 dist-test-slave-n106 krb5kdc[16492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265995, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.61@KRBTEST.COM for kudu/127.25.109.61@KRBTEST.COM
I20260924 16:06:35.977542 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:35.967130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:36745 (local address 127.0.0.1:43446)
0924 16:06:35.967712 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:35.967736 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:35.967833 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:35.968103 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:35.968108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:35.968305 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:35.968683 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:35.968695 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:35.969529 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:35.969534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:35.970428 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:35.970441 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:35.970888 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:35.970894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:35.971029 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:35.971490 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:35.971522 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:35.974664 (+ 3142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:35.976149 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:35.976152 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:35.976153 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:35.976332 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:35.976646 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:35.976648 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:35.976649 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:35.976723 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:35.977096 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:35.977098 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:35.977172 (+ 74us) client_negotiation.cc:769] Sending connection context
0924 16:06:35.977254 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 16:06:35.977344 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 16:06:35.977669 16652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:35.967217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43446 (local address 127.25.109.61:36745)
0924 16:06:35.967496 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:35.967499 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:35.967808 (+ 309us) server_negotiation.cc:407] Connection header received
0924 16:06:35.967927 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:35.967930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:35.967974 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:35.968052 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:35.968834 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:35.969415 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:35.970546 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:35.970774 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:35.970847 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:35.974837 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:35.974840 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:35.974842 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:35.974862 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:35.976057 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:35.976464 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:35.976468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:35.976470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:35.976558 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:35.976804 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:35.976806 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:35.976807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:35.977010 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:35.977084 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:35.977309 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 16:06:35.977436 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":96,"threads_started":1}
I20260924 16:06:35.978304 16650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.60" port: 39015 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.109.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.60.keytab.
Entry for principal kudu/127.25.109.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.60.keytab.
I20260924 16:06:35.979022 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:35.978745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:39015 (local address 127.0.0.1:58766)
0924 16:06:35.978860 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:35.978937 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:39015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 16:06:35.979247 16601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.60:39015: connect: Connection refused (error 111)
I20260924 16:06:35.979559 26036 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
/tmp/dist-test-taskCcKW0d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.60:39015
--webserver_interface=127.25.109.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.60.keytab
--principal=kudu/127.25.109.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.109.20:34671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.62:44327,127.25.109.61:36745,127.25.109.60:39015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:06:35.980247 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:35.979973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:39015 (local address 127.0.0.1:58770)
0924 16:06:35.980109 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:35.980169 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:39015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 16:06:35.980432 16650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.109.60:39015: Network error: Client connection negotiation failed: client connection to 127.25.109.60:39015: connect: Connection refused (error 111)
I20260924 16:06:36.000227 16574 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.109.60" port: 39015 } attempt: 1
I20260924 16:06:36.001025 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:36.000740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:39015 (local address 127.0.0.1:58774)
0924 16:06:36.000881 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:36.000947 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:39015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 16:06:36.001766 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:36.001563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:39015 (local address 127.0.0.1:58778)
0924 16:06:36.001670 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:36.001713 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:39015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 16:06:36.001967 16574 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.109.60:39015: Network error: Client connection negotiation failed: client connection to 127.25.109.60:39015: connect: Connection refused (error 111)
I20260924 16:06:36.029716 16650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.109.60" port: 39015 } attempt: 1
I20260924 16:06:36.030651 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:36.030277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:39015 (local address 127.0.0.1:58784)
0924 16:06:36.030476 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:36.030557 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:39015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 16:06:36.031383 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:36.031161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:39015 (local address 127.0.0.1:58790)
0924 16:06:36.031264 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:36.031319 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:39015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 16:06:36.031527 16650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.109.60:39015: Network error: Client connection negotiation failed: client connection to 127.25.109.60:39015: connect: Connection refused (error 111)
I20260924 16:06:36.077140 16574 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.109.60" port: 39015 } attempt: 2
I20260924 16:06:36.077992 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:36.077700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:39015 (local address 127.0.0.1:58802)
0924 16:06:36.077849 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:36.077913 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:39015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 16:06:36.078722 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:36.078493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:39015 (local address 127.0.0.1:58808)
0924 16:06:36.078620 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:36.078665 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:39015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 16:06:36.078850 16574 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.109.60:39015: Network error: Client connection negotiation failed: client connection to 127.25.109.60:39015: connect: Connection refused (error 111)
W20260924 16:06:36.090619 16658 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 16:06:36.090927 16658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:36.090986 16658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:36.091089 16658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:36.095112 16658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:36.095163 16658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:36.095182 16658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:36.095238 16658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:36.095273 16658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:36.099694 16650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.109.60" port: 39015 } attempt: 2
I20260924 16:06:36.100525 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:36.100202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:39015 (local address 127.0.0.1:58818)
0924 16:06:36.100349 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:36.100450 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:39015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 16:06:36.101233 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:36.101007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:39015 (local address 127.0.0.1:58830)
0924 16:06:36.101126 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:36.101175 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.60:39015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 16:06:36.101357 16650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.109.60:39015: Network error: Client connection negotiation failed: client connection to 127.25.109.60:39015: connect: Connection refused (error 111)
I20260924 16:06:36.101311 16658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.20:34671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.109.62:44327,127.25.109.61:36745,127.25.109.60:39015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/krb5kdc/kudu_127.25.109.60.keytab
--principal=kudu/127.25.109.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.60:39015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.109.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:06:36.102430 16658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:36.103528 16658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:36.111949 16664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:36.111956 16663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:36.111956 16666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:36.112111 16658 server_base.cc:1034] running on GCE node
I20260924 16:06:36.112789 16658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:36.113796 16658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:36.114944 16658 hybrid_clock.cc:648] HybridClock initialized: now 1790265996114906 us; error 48 us; skew 500 ppm
Sep 24 16:06:36 dist-test-slave-n106 krb5kdc[16492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265996, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:36.117573 16658 init.cc:377] Logged in from keytab as kudu/127.25.109.60@KRBTEST.COM (short username kudu)
I20260924 16:06:36.118341 16658 webserver.cc:460] Webserver started at http://127.25.109.60:42507/ using document root <none> and password file <none>
I20260924 16:06:36.118837 16658 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:36.118893 16658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:36.119052 16658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:36.121044 16658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-2/data/instance:
uuid: "531228f7d6fe4c5b9124e2f2b2824802"
format_stamp: "Formatted at 2026-09-24 16:06:36 on dist-test-slave-n106"
server_key: "e3bc1a127ae99bc7a01d68a79f8bbc23"
server_key_iv: "86707a94c130cf13d21966cebd628541"
server_key_version: "encryptionkey@0"
I20260924 16:06:36.121544 16658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-2/wal/instance:
uuid: "531228f7d6fe4c5b9124e2f2b2824802"
format_stamp: "Formatted at 2026-09-24 16:06:36 on dist-test-slave-n106"
server_key: "e3bc1a127ae99bc7a01d68a79f8bbc23"
server_key_iv: "86707a94c130cf13d21966cebd628541"
server_key_version: "encryptionkey@0"
I20260924 16:06:36.125130 16658 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:06:36.127456 16673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:36.128741 16658 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:06:36.128876 16658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-2/data,/tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-2/wal
uuid: "531228f7d6fe4c5b9124e2f2b2824802"
format_stamp: "Formatted at 2026-09-24 16:06:36 on dist-test-slave-n106"
server_key: "e3bc1a127ae99bc7a01d68a79f8bbc23"
server_key_iv: "86707a94c130cf13d21966cebd628541"
server_key_version: "encryptionkey@0"
I20260924 16:06:36.128989 16658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:36.146575 16658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:36.149331 16658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:36.149526 16658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:36.156000 16658 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.60:39015
I20260924 16:06:36.156302 16726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.60:39015 every 8 connection(s)
I20260924 16:06:36.156991 16658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-2/data/info.pb
I20260924 16:06:36.159694 16727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:36.163697 16727 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44327 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:06:36.165434 26036 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu as pid 16658
I20260924 16:06:36.165621 26036 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCcKW0d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265708917108-26036-0/minicluster-data/master-2/wal/instance
I20260924 16:06:36.165935 26036 external_mini_cluster.cc:1428] Setting key c996303850c3b1ed8a37428db5a19609
Sep 24 16:06:36 dist-test-slave-n106 krb5kdc[16492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265996, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.60@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
Sep 24 16:06:36 dist-test-slave-n106 krb5kdc[16492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.60@KRBTEST.COM
I20260924 16:06:36.177995 16574 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.109.60" port: 39015 } attempt: 3
I20260924 16:06:36.178460 16577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:36.165690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46410 (local address 127.25.109.62:44327)
0924 16:06:36.165840 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:36.165846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:36.166737 (+ 891us) server_negotiation.cc:407] Connection header received
0924 16:06:36.167724 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:36.167727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:36.167780 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:36.167872 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:36.169012 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.169802 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.171137 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.171462 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.171568 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:36.174405 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:36.174409 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:36.174411 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:36.174437 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:36.176085 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:36.176727 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:36.176731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:36.176735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:36.176825 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:36.177214 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:36.177218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:36.177220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:36.177474 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:36.177553 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:36.178082 (+ 529us) server_negotiation.cc:299] Negotiation successful
0924 16:06:36.178281 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:06:36.179473 16729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:36.166101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.62:44327 (local address 127.0.0.1:46410)
0924 16:06:36.166571 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:36.166629 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:36.167526 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:36.167953 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:36.167959 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:36.168483 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:36.168841 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:36.168864 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.169912 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.169914 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:36.170992 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:36.171016 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.171604 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.171606 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:36.171751 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:36.172347 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:36.172374 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:36.174234 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:36.176199 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:36.176208 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:36.176219 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:36.176630 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:36.176923 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:36.176926 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:36.176928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:36.177111 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:36.177576 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:36.177581 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:36.177829 (+ 248us) client_negotiation.cc:769] Sending connection context
0924 16:06:36.177994 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 16:06:36.178326 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"spinlock_wait_cycles":8064,"thread_start_us":135,"threads_started":1}
I20260924 16:06:36.179473 16731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:36.166560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58838 (local address 127.25.109.60:39015)
0924 16:06:36.167004 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:36.167018 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:36.167060 (+ 42us) server_negotiation.cc:407] Connection header received
0924 16:06:36.167541 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:36.167562 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:36.167817 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:36.168004 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:36.168917 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.170128 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.171259 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.171775 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.171851 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:36.174816 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:36.174824 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:36.174840 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:36.174860 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:36.176459 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:36.176930 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:36.176936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:36.176937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:36.177100 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:36.177393 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:36.177396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:36.177397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:36.177829 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:36.178884 (+ 1055us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:36.178981 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 16:06:36.179121 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":358,"spinlock_wait_cycles":6400,"thread_start_us":136,"threads_started":1}
I20260924 16:06:36.180838 16727 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.61" port: 36745 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:06:36 dist-test-slave-n106 krb5kdc[16492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265995, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.62@KRBTEST.COM for kudu/127.25.109.60@KRBTEST.COM
Sep 24 16:06:36 dist-test-slave-n106 krb5kdc[16492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265996, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.60@KRBTEST.COM for kudu/127.25.109.61@KRBTEST.COM
Sep 24 16:06:36 dist-test-slave-n106 krb5kdc[16492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.62@KRBTEST.COM
I20260924 16:06:36.189836 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:36.178385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:39015 (local address 127.0.0.1:58840)
0924 16:06:36.178658 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:36.178672 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:36.178741 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:36.179245 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:36.179248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:36.179404 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:36.179478 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:36.179483 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.180536 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.180540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:36.181182 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:36.181197 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.182571 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.182575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:36.182667 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:36.183175 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:36.183189 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:36.184709 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:36.188123 (+ 3414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:36.188126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:36.188129 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:36.188389 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:36.188759 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:36.188763 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:36.188765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:36.188861 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:36.189295 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:36.189298 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:36.189399 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 16:06:36.189522 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 16:06:36.189652 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192}
I20260924 16:06:36.189847 16732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:36.178653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58840 (local address 127.25.109.60:39015)
0924 16:06:36.179043 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:36.179050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:36.179060 (+ 10us) server_negotiation.cc:407] Connection header received
0924 16:06:36.179088 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:36.179090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:36.179123 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:36.179227 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:36.179598 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.180367 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.182130 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.182454 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.182555 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:36.184838 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:36.184841 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:36.184843 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:36.184865 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:36.187977 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:36.188547 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:36.188553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:36.188557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:36.188646 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:36.188965 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:36.188969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:36.188971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:36.189188 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:36.189305 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:36.189573 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 16:06:36.189664 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"spinlock_wait_cycles":39296,"thread_start_us":96,"threads_started":1}
I20260924 16:06:36.190383 16652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:36.181503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43462 (local address 127.25.109.61:36745)
0924 16:06:36.181646 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:36.181651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:36.181667 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:06:36.181710 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:36.181713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:36.181758 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:36.181843 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:36.182227 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.182925 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.184064 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.184475 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.184552 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:36.186835 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:36.186838 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:36.186843 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:36.186873 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:36.188523 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:36.189012 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:36.189016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:36.189018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:36.189102 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:36.189439 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:36.189442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:36.189444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:36.189669 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:36.189766 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:36.190079 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 16:06:36.190227 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:06:36.191072 16729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:36.181302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.61:36745 (local address 127.0.0.1:43462)
0924 16:06:36.181417 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:36.181432 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:36.181502 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:36.181864 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:36.181866 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:36.181983 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:36.182063 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:36.182067 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.183057 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.183060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:36.183925 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:36.183936 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.184720 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.184723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:36.184838 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:36.185296 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:36.185310 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:36.186699 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:36.188631 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:36.188634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:36.188636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:36.188884 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:36.189234 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:36.189238 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:36.189241 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:36.189336 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:36.189779 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:36.189782 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:36.189874 (+ 92us) client_negotiation.cc:769] Sending connection context
0924 16:06:36.190791 (+ 917us) client_negotiation.cc:240] Negotiation successful
0924 16:06:36.190940 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 16:06:36.191953 16727 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.109.60" port: 39015 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:06:36.192787 16650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.109.60" port: 39015 } attempt: 3
I20260924 16:06:36.193571 16574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4eaea955d16422a82c9fe17eb97dc54: Bootstrap starting.
I20260924 16:06:36.196455 16574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4eaea955d16422a82c9fe17eb97dc54: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:36.197393 16574 log.cc:824] T 00000000000000000000000000000000 P b4eaea955d16422a82c9fe17eb97dc54: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:06:36 dist-test-slave-n106 krb5kdc[16492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265996, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.60@KRBTEST.COM for kudu/127.25.109.60@KRBTEST.COM
I20260924 16:06:36.197916 16577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:36.182885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46414 (local address 127.25.109.62:44327)
0924 16:06:36.183014 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:36.183020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:36.183035 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:06:36.183274 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:36.183277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:36.183323 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:36.183416 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:36.183989 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.184543 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.185486 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.185857 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.185945 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:36.188881 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:36.188884 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:36.188887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:36.188913 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:36.191302 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:36.191763 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:36.191766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:36.191767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:36.191856 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:36.192175 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:36.192179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:36.192181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:36.192427 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:36.192640 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:36.193360 (+ 720us) server_negotiation.cc:299] Negotiation successful
0924 16:06:36.193485 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 16:06:36 dist-test-slave-n106 krb5kdc[16492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265995, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.109.61@KRBTEST.COM for kudu/127.25.109.60@KRBTEST.COM
I20260924 16:06:36.199973 16574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4eaea955d16422a82c9fe17eb97dc54: No bootstrap required, opened a new log
I20260924 16:06:36.202251 16729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:36.192472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:39015 (local address 127.0.0.1:58842)
0924 16:06:36.192575 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:36.192589 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:36.192662 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:36.193082 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:36.193085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:36.193244 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:36.193340 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:36.193347 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.194248 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.194251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:36.194740 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:36.194747 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.195852 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.195855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:36.195911 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:36.196728 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:36.196743 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:36.198331 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:36.200712 (+ 2381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:36.200715 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:36.200716 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:36.201112 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:36.201367 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:36.201369 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:36.201371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:36.201438 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:36.201817 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:36.201819 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:36.201888 (+ 69us) client_negotiation.cc:769] Sending connection context
0924 16:06:36.201977 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 16:06:36.202109 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 16:06:36.202817 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:36.193156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.60:39015 (local address 127.0.0.1:58850)
0924 16:06:36.193372 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:36.193386 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:36.193456 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:36.193922 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:36.193925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:36.194087 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:36.194177 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:36.194184 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.195184 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.195187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:36.195730 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:36.195738 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.196340 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.196344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:36.196477 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:36.196923 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:36.196938 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:36.198841 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:36.201289 (+ 2448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:36.201293 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:36.201295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:36.201523 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:36.201849 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:36.201852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:36.201854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:36.201939 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:36.202336 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:36.202339 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:36.202424 (+ 85us) client_negotiation.cc:769] Sending connection context
0924 16:06:36.202501 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 16:06:36.202637 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260924 16:06:36.202819 16732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:36.192711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58842 (local address 127.25.109.60:39015)
0924 16:06:36.192828 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:36.192832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:36.192844 (+ 12us) server_negotiation.cc:407] Connection header received
0924 16:06:36.192878 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:36.192880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:36.192919 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:36.192993 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:36.193447 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.194140 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.194849 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.195084 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.195189 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:36.198452 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:36.198456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:36.198458 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:36.198484 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:36.200602 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:36.201213 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:36.201216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:36.201218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:36.201290 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:36.201519 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:36.201522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:36.201524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:36.201701 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:36.201787 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:36.202572 (+ 785us) server_negotiation.cc:299] Negotiation successful
0924 16:06:36.202680 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 16:06:36.203586 16731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:36.193613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58850 (local address 127.25.109.60:39015)
0924 16:06:36.193714 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:36.193718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:36.193731 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:06:36.193763 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:36.193765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:36.193801 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:36.194284 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:36.194363 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.195072 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.195847 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.196172 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.196332 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:36.198992 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:36.198998 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:36.199003 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:36.199027 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:36.201178 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:36.201659 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:36.201664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:36.201665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:36.201748 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:36.202048 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:36.202051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:36.202053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:36.202231 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:36.203080 (+ 849us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:36.203158 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 16:06:36.203268 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 16:06:36 dist-test-slave-n106 krb5kdc[16492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.109.61@KRBTEST.COM
I20260924 16:06:36.206524 16650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 183d364a616c4b37857e40285941660d: Bootstrap starting.
I20260924 16:06:36.206538 16727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 531228f7d6fe4c5b9124e2f2b2824802: Bootstrap starting.
I20260924 16:06:36.209134 16650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 183d364a616c4b37857e40285941660d: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:36.209133 16727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 531228f7d6fe4c5b9124e2f2b2824802: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:36.209658 16652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:36.199371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43468 (local address 127.25.109.61:36745)
0924 16:06:36.199464 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:36.199468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:36.199562 (+ 94us) server_negotiation.cc:407] Connection header received
0924 16:06:36.199751 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:36.199754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:36.199801 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:36.199858 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:36.200824 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.201579 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.202758 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.203086 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.203170 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:36.206121 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:36.206125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:36.206127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:36.206152 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:36.207834 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:36.208312 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:36.208316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:36.208317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:36.208401 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:36.208713 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:36.208716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:36.208718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:36.208916 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:36.209086 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:36.209420 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 16:06:36.209527 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 16:06:36.209993 16650 log.cc:824] T 00000000000000000000000000000000 P 183d364a616c4b37857e40285941660d: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:36.210021 16727 log.cc:824] T 00000000000000000000000000000000 P 531228f7d6fe4c5b9124e2f2b2824802: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:36.210754 16574 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4eaea955d16422a82c9fe17eb97dc54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4eaea955d16422a82c9fe17eb97dc54" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44327 } } peers { permanent_uuid: "183d364a616c4b37857e40285941660d" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 36745 } } peers { permanent_uuid: "531228f7d6fe4c5b9124e2f2b2824802" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 39015 } }
I20260924 16:06:36.211018 16574 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4eaea955d16422a82c9fe17eb97dc54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4eaea955d16422a82c9fe17eb97dc54, State: Initialized, Role: FOLLOWER
I20260924 16:06:36.211373 16574 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4eaea955d16422a82c9fe17eb97dc54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4eaea955d16422a82c9fe17eb97dc54" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44327 } } peers { permanent_uuid: "183d364a616c4b37857e40285941660d" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 36745 } } peers { permanent_uuid: "531228f7d6fe4c5b9124e2f2b2824802" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 39015 } }
I20260924 16:06:36.212538 16727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 531228f7d6fe4c5b9124e2f2b2824802: No bootstrap required, opened a new log
I20260924 16:06:36.212663 16650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 183d364a616c4b37857e40285941660d: No bootstrap required, opened a new log
I20260924 16:06:36.212793 16574 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4eaea955d16422a82c9fe17eb97dc54 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:36.213182 26036 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:06:36.214396 16548 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b4eaea955d16422a82c9fe17eb97dc54"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46414
I20260924 16:06:36.214610 16548 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4eaea955d16422a82c9fe17eb97dc54 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 16:06:36.214725 16548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4eaea955d16422a82c9fe17eb97dc54 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:36.215960 16548 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4eaea955d16422a82c9fe17eb97dc54 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4eaea955d16422a82c9fe17eb97dc54" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44327 } } peers { permanent_uuid: "183d364a616c4b37857e40285941660d" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 36745 } } peers { permanent_uuid: "531228f7d6fe4c5b9124e2f2b2824802" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 39015 } }
I20260924 16:06:36.216683 16650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 183d364a616c4b37857e40285941660d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4eaea955d16422a82c9fe17eb97dc54" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44327 } } peers { permanent_uuid: "183d364a616c4b37857e40285941660d" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 36745 } } peers { permanent_uuid: "531228f7d6fe4c5b9124e2f2b2824802" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 39015 } }
I20260924 16:06:36.216627 16727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 531228f7d6fe4c5b9124e2f2b2824802 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4eaea955d16422a82c9fe17eb97dc54" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44327 } } peers { permanent_uuid: "183d364a616c4b37857e40285941660d" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 36745 } } peers { permanent_uuid: "531228f7d6fe4c5b9124e2f2b2824802" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 39015 } }
I20260924 16:06:36.216886 16650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 183d364a616c4b37857e40285941660d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 183d364a616c4b37857e40285941660d, State: Initialized, Role: FOLLOWER
I20260924 16:06:36.216886 16727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 531228f7d6fe4c5b9124e2f2b2824802 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 531228f7d6fe4c5b9124e2f2b2824802, State: Initialized, Role: FOLLOWER
I20260924 16:06:36.217018 16548 leader_election.cc:288] T 00000000000000000000000000000000 P b4eaea955d16422a82c9fe17eb97dc54 [CANDIDATE]: Term 1 election: Requested vote from peers 183d364a616c4b37857e40285941660d (127.25.109.61:36745), 531228f7d6fe4c5b9124e2f2b2824802 (127.25.109.60:39015)
I20260924 16:06:36.217200 16650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 183d364a616c4b37857e40285941660d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4eaea955d16422a82c9fe17eb97dc54" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44327 } } peers { permanent_uuid: "183d364a616c4b37857e40285941660d" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 36745 } } peers { permanent_uuid: "531228f7d6fe4c5b9124e2f2b2824802" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 39015 } }
I20260924 16:06:36.217214 16727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 531228f7d6fe4c5b9124e2f2b2824802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4eaea955d16422a82c9fe17eb97dc54" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44327 } } peers { permanent_uuid: "183d364a616c4b37857e40285941660d" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 36745 } } peers { permanent_uuid: "531228f7d6fe4c5b9124e2f2b2824802" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 39015 } }
I20260924 16:06:36.218473 16727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 531228f7d6fe4c5b9124e2f2b2824802 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:36.218528 16700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b4eaea955d16422a82c9fe17eb97dc54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "531228f7d6fe4c5b9124e2f2b2824802"
I20260924 16:06:36.218649 16650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 183d364a616c4b37857e40285941660d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:36.218708 16700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 531228f7d6fe4c5b9124e2f2b2824802 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:36.218863 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b4eaea955d16422a82c9fe17eb97dc54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "183d364a616c4b37857e40285941660d"
I20260924 16:06:36.219062 16624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 183d364a616c4b37857e40285941660d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:36.220170 16700 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 531228f7d6fe4c5b9124e2f2b2824802 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4eaea955d16422a82c9fe17eb97dc54 in term 1.
I20260924 16:06:36.220458 16624 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 183d364a616c4b37857e40285941660d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4eaea955d16422a82c9fe17eb97dc54 in term 1.
I20260924 16:06:36.220785 16525 leader_election.cc:302] T 00000000000000000000000000000000 P b4eaea955d16422a82c9fe17eb97dc54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 531228f7d6fe4c5b9124e2f2b2824802, b4eaea955d16422a82c9fe17eb97dc54; no voters:
I20260924 16:06:36.226457 16505 master_runner.cc:263] No leader master found from master b4eaea955d16422a82c9fe17eb97dc54
I20260924 16:06:36.226528 16505 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 16:06:36.226854 16748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 183d364a616c4b37857e40285941660d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4eaea955d16422a82c9fe17eb97dc54" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44327 } } peers { permanent_uuid: "183d364a616c4b37857e40285941660d" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 36745 } } peers { permanent_uuid: "531228f7d6fe4c5b9124e2f2b2824802" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 39015 } } }
I20260924 16:06:36.226974 16748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 183d364a616c4b37857e40285941660d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:06:36.228766 16658 master_runner.cc:263] No leader master found from master 531228f7d6fe4c5b9124e2f2b2824802
I20260924 16:06:36.228830 16658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 16:06:36.228833 16736 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4eaea955d16422a82c9fe17eb97dc54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4eaea955d16422a82c9fe17eb97dc54" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44327 } } peers { permanent_uuid: "183d364a616c4b37857e40285941660d" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 36745 } } peers { permanent_uuid: "531228f7d6fe4c5b9124e2f2b2824802" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 39015 } } }
I20260924 16:06:36.228976 16736 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4eaea955d16422a82c9fe17eb97dc54 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:06:36.229072 16736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4eaea955d16422a82c9fe17eb97dc54 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:36.229147 16736 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4eaea955d16422a82c9fe17eb97dc54 [term 1 LEADER]: Becoming Leader. State: Replica: b4eaea955d16422a82c9fe17eb97dc54, State: Running, Role: LEADER
I20260924 16:06:36.229502 16736 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4eaea955d16422a82c9fe17eb97dc54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4eaea955d16422a82c9fe17eb97dc54" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44327 } } peers { permanent_uuid: "183d364a616c4b37857e40285941660d" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 36745 } } peers { permanent_uuid: "531228f7d6fe4c5b9124e2f2b2824802" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 39015 } }
I20260924 16:06:36.230291 16582 master_runner.cc:263] No leader master found from master 183d364a616c4b37857e40285941660d
I20260924 16:06:36.230352 16582 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 16:06:36.231552 16736 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4eaea955d16422a82c9fe17eb97dc54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4eaea955d16422a82c9fe17eb97dc54. Latest consensus state: current_term: 1 leader_uuid: "b4eaea955d16422a82c9fe17eb97dc54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4eaea955d16422a82c9fe17eb97dc54" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44327 } } peers { permanent_uuid: "183d364a616c4b37857e40285941660d" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 36745 } } peers { permanent_uuid: "531228f7d6fe4c5b9124e2f2b2824802" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 39015 } } }
I20260924 16:06:36.231676 16736 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4eaea955d16422a82c9fe17eb97dc54 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:36.233218 16652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:36.223269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43472 (local address 127.25.109.61:36745)
0924 16:06:36.223378 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:36.223382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:36.223834 (+ 452us) server_negotiation.cc:407] Connection header received
0924 16:06:36.223973 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:36.223976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:36.224020 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:36.224107 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:36.225055 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.225768 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.227241 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.227551 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.227627 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:36.229613 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:36.229617 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:36.229620 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:36.229647 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:36.231145 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:36.231655 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:36.231659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:36.231661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:36.231774 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:36.232210 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:36.232214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:36.232216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:36.232471 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:36.232594 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:36.232974 (+ 380us) server_negotiation.cc:299] Negotiation successful
0924 16:06:36.233111 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 16:06:36.233929 16731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:36.223279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58860 (local address 127.25.109.60:39015)
0924 16:06:36.223377 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:36.223382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:36.224695 (+ 1313us) server_negotiation.cc:407] Connection header received
0924 16:06:36.224872 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:36.224875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:36.224918 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:36.224966 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:36.225929 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.226603 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.227803 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.228120 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.228191 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:36.230158 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:36.230164 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:36.230167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:36.230195 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:36.232067 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:36.232602 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:36.232606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:36.232608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:36.232706 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:36.233011 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:36.233015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:36.233016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:36.233241 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:36.233348 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:36.233669 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 16:06:36.233787 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 16:06:36.234458 16769 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 183d364a616c4b37857e40285941660d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:06:36.234542 16769 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 16:06:36.237329 16747 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b4eaea955d16422a82c9fe17eb97dc54: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:06:36.237414 16747 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 16:06:36.237895 16771 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 531228f7d6fe4c5b9124e2f2b2824802: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:06:36.237977 16771 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:06:36.240700 16781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:36.241761 16781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:36.242492 16577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:36.224321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46422 (local address 127.25.109.62:44327)
0924 16:06:36.224475 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:36.224480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:36.233896 (+ 9416us) server_negotiation.cc:407] Connection header received
0924 16:06:36.234072 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:36.234075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:36.234134 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:36.234184 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:36.235015 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.235704 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.236555 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:36.236773 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:36.236880 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:36.238634 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:36.238638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:36.238640 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:36.238670 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:36.240436 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:36.241024 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:36.241028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:36.241030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:36.241116 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:36.241439 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:36.241443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:36.241444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:36.241663 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:36.241782 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:36.242224 (+ 442us) server_negotiation.cc:299] Negotiation successful
0924 16:06:36.242351 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 16:06:36.244029 16749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 531228f7d6fe4c5b9124e2f2b2824802 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4eaea955d16422a82c9fe17eb97dc54" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44327 } } peers { permanent_uuid: "183d364a616c4b37857e40285941660d" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 36745 } } peers { permanent_uuid: "531228f7d6fe4c5b9124e2f2b2824802" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 39015 } } }
I20260924 16:06:36.244181 16749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 531228f7d6fe4c5b9124e2f2b2824802 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:06:36.247973 16700 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 531228f7d6fe4c5b9124e2f2b2824802 [term 1 FOLLOWER]: Refusing update from remote peer b4eaea955d16422a82c9fe17eb97dc54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:06:36.248929 16624 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 183d364a616c4b37857e40285941660d [term 1 FOLLOWER]: Refusing update from remote peer b4eaea955d16422a82c9fe17eb97dc54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:06:36.249650 16772 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b4eaea955d16422a82c9fe17eb97dc54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "183d364a616c4b37857e40285941660d" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:06:36.251468 16772 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b4eaea955d16422a82c9fe17eb97dc54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "531228f7d6fe4c5b9124e2f2b2824802" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 39015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:06:36.255360 16749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 531228f7d6fe4c5b9124e2f2b2824802 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4eaea955d16422a82c9fe17eb97dc54. Latest consensus state: current_term: 1 leader_uuid: "b4eaea955d16422a82c9fe17eb97dc54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4eaea955d16422a82c9fe17eb97dc54" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44327 } } peers { permanent_uuid: "183d364a616c4b37857e40285941660d" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 36745 } } peers { permanent_uuid: "531228f7d6fe4c5b9124e2f2b2824802" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 39015 } } }
I20260924 16:06:36.255484 16749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 531228f7d6fe4c5b9124e2f2b2824802 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:06:36.259007 16748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 183d364a616c4b37857e40285941660d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4eaea955d16422a82c9fe17eb97dc54. Latest consensus state: current_term: 1 leader_uuid: "b4eaea955d16422a82c9fe17eb97dc54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4eaea955d16422a82c9fe17eb97dc54" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44327 } } peers { permanent_uuid: "183d364a616c4b37857e40285941660d" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 36745 } } peers { permanent_uuid: "531228f7d6fe4c5b9124e2f2b2824802" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 39015 } } }
I20260924 16:06:36.259133 16748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 183d364a616c4b37857e40285941660d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:06:36.266630 16772 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4eaea955d16422a82c9fe17eb97dc54 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4eaea955d16422a82c9fe17eb97dc54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4eaea955d16422a82c9fe17eb97dc54" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44327 } } peers { permanent_uuid: "183d364a616c4b37857e40285941660d" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 36745 } } peers { permanent_uuid: "531228f7d6fe4c5b9124e2f2b2824802" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 39015 } } }
I20260924 16:06:36.266762 16772 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4eaea955d16422a82c9fe17eb97dc54 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:36.266886 16785 mvcc.cc:205] Tried to move back new op lower bound from 7332929520622297088 to 7332929520560082944. Current Snapshot: MvccSnapshot[applied={T|T < 7332929520622297088}]
I20260924 16:06:36.268041 16748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 183d364a616c4b37857e40285941660d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4eaea955d16422a82c9fe17eb97dc54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4eaea955d16422a82c9fe17eb97dc54" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44327 } } peers { permanent_uuid: "183d364a616c4b37857e40285941660d" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 36745 } } peers { permanent_uuid: "531228f7d6fe4c5b9124e2f2b2824802" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 39015 } } }
I20260924 16:06:36.268162 16748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 183d364a616c4b37857e40285941660d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:06:36.271107 16788 mvcc.cc:205] Tried to move back new op lower bound from 7332929520622297088 to 7332929520560082944. Current Snapshot: MvccSnapshot[applied={T|T < 7332929520622297088}]
I20260924 16:06:36.271761 16736 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4eaea955d16422a82c9fe17eb97dc54 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4eaea955d16422a82c9fe17eb97dc54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4eaea955d16422a82c9fe17eb97dc54" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44327 } } peers { permanent_uuid: "183d364a616c4b37857e40285941660d" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 36745 } } peers { permanent_uuid: "531228f7d6fe4c5b9124e2f2b2824802" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 39015 } } }
I20260924 16:06:36.271875 16736 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4eaea955d16422a82c9fe17eb97dc54 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:36.272716 16749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 531228f7d6fe4c5b9124e2f2b2824802 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4eaea955d16422a82c9fe17eb97dc54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4eaea955d16422a82c9fe17eb97dc54" member_type: VOTER last_known_addr { host: "127.25.109.62" port: 44327 } } peers { permanent_uuid: "183d364a616c4b37857e40285941660d" member_type: VOTER last_known_addr { host: "127.25.109.61" port: 36745 } } peers { permanent_uuid: "531228f7d6fe4c5b9124e2f2b2824802" member_type: VOTER last_known_addr { host: "127.25.109.60" port: 39015 } } }
I20260924 16:06:36.272841 16749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 531228f7d6fe4c5b9124e2f2b2824802 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:06:36.275489 16781 catalog_manager.cc:1347] Generated new cluster ID: bd46ea81a61b44f5883f941f6dd2fb5e
I20260924 16:06:36.275557 16781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:36.275135 16786 mvcc.cc:205] Tried to move back new op lower bound from 7332929520622297088 to 7332929520560082944. Current Snapshot: MvccSnapshot[applied={T|T < 7332929520622297088}]
I20260924 16:06:36.295601 16781 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:36.296957 16781 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:36.327977 16781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4eaea955d16422a82c9fe17eb97dc54: Generated new TSK 0
I20260924 16:06:36.328936 16781 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 16:06:36.365713 16577 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:36.362907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46432 (local address 127.25.109.62:44327)
0924 16:06:36.363042 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:36.363047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:36.364607 (+ 1560us) server_negotiation.cc:407] Connection header received
0924 16:06:36.364825 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:36.364829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:36.364882 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:36.364937 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:36.365570 (+ 633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46432: BlockingRecv error: recv got EOF from 127.0.0.1:46432 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 16:06:36.367065 16652 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:36.363358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43476 (local address 127.25.109.61:36745)
0924 16:06:36.363498 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:36.363503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:36.366203 (+ 2700us) server_negotiation.cc:407] Connection header received
0924 16:06:36.366374 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:36.366377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:36.366426 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:36.366475 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:36.366947 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43476: BlockingRecv error: recv got EOF from 127.0.0.1:43476 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 16:06:36.368362 16731 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:36.363805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58862 (local address 127.25.109.60:39015)
0924 16:06:36.363920 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:36.363924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:36.367515 (+ 3591us) server_negotiation.cc:407] Connection header received
0924 16:06:36.367680 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:36.367684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:36.367733 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:36.367782 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:36.368250 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58862: BlockingRecv error: recv got EOF from 127.0.0.1:58862 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260924 16:06:36.372546 16652 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:36.371098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43482 (local address 127.25.109.61:36745)
0924 16:06:36.371207 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:36.371211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:36.371603 (+ 392us) server_negotiation.cc:407] Connection header received
0924 16:06:36.371804 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:36.371807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:36.371879 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:36.371930 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:36.372445 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43482: BlockingRecv error: recv got EOF from 127.0.0.1:43482 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 16:06:36.378527 16652 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:36.377113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43486 (local address 127.25.109.61:36745)
0924 16:06:36.377209 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:36.377213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:36.377698 (+ 485us) server_negotiation.cc:407] Connection header received
0924 16:06:36.377873 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:36.377876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:36.377923 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:36.377970 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:36.378438 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43486: BlockingRecv error: recv got EOF from 127.0.0.1:43486 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 16:06:36.384227 16652 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:36.382701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43490 (local address 127.25.109.61:36745)
0924 16:06:36.382800 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:36.382804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:36.383337 (+ 533us) server_negotiation.cc:407] Connection header received
0924 16:06:36.383549 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:36.383552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:36.383600 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:36.383650 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:36.384128 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43490: BlockingRecv error: recv got EOF from 127.0.0.1:43490 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 16:06:36.392349 16652 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:36.390564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43506 (local address 127.25.109.61:36745)
0924 16:06:36.390715 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:36.390719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:36.391393 (+ 674us) server_negotiation.cc:407] Connection header received
0924 16:06:36.391602 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:36.391606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:36.391660 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:36.391714 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:36.392240 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43506: BlockingRecv error: recv got EOF from 127.0.0.1:43506 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260924 16:06:36.405232 16652 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:36.403368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43518 (local address 127.25.109.61:36745)
0924 16:06:36.403511 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:36.403515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:36.404136 (+ 621us) server_negotiation.cc:407] Connection header received
0924 16:06:36.404369 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:36.404372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:36.404459 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:36.404519 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:36.405099 (+ 580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43518: BlockingRecv error: recv got EOF from 127.0.0.1:43518 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 16:06:36.428432 16652 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:36.426002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43530 (local address 127.25.109.61:36745)
0924 16:06:36.426228 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:36.426234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:36.426998 (+ 764us) server_negotiation.cc:407] Connection header received
0924 16:06:36.427321 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:36.427325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:36.427539 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:36.427628 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:36.428291 (+ 663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43530: BlockingRecv error: recv got EOF from 127.0.0.1:43530 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
W20260924 16:06:36.464380 16652 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:36.463082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43534 (local address 127.25.109.61:36745)
0924 16:06:36.463253 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:36.463258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:36.463433 (+ 175us) server_negotiation.cc:407] Connection header received
0924 16:06:36.463629 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:36.463634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:36.463691 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:36.463785 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:36.464255 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43534: BlockingRecv error: recv got EOF from 127.0.0.1:43534 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260924 16:06:36.531921 16652 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:36.530669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43550 (local address 127.25.109.61:36745)
0924 16:06:36.530831 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:36.530836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:36.530972 (+ 136us) server_negotiation.cc:407] Connection header received
0924 16:06:36.531165 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:36.531168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:36.531216 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:36.531309 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:36.531798 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43550: BlockingRecv error: recv got EOF from 127.0.0.1:43550 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 16:06:36.663148 16652 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:36.662092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43560 (local address 127.25.109.61:36745)
0924 16:06:36.662247 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:36.662251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:36.662340 (+ 89us) server_negotiation.cc:407] Connection header received
0924 16:06:36.662466 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:36.662469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:36.662519 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:36.662613 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:36.662999 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43560: BlockingRecv error: recv got EOF from 127.0.0.1:43560 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260924 16:06:36.922767 16652 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:36.921518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43574 (local address 127.25.109.61:36745)
0924 16:06:36.921657 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:36.921661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:36.921845 (+ 184us) server_negotiation.cc:407] Connection header received
0924 16:06:36.922031 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:36.922034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:36.922084 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:36.922182 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:36.922642 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43574: BlockingRecv error: recv got EOF from 127.0.0.1:43574 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 16:06:37.235811 16769 catalog_manager.cc:1259] Loaded cluster ID: bd46ea81a61b44f5883f941f6dd2fb5e
I20260924 16:06:37.235895 16769 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 183d364a616c4b37857e40285941660d: loading cluster ID for follower catalog manager: success
I20260924 16:06:37.237651 16769 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 183d364a616c4b37857e40285941660d: acquiring CA information for follower catalog manager: success
I20260924 16:06:37.239091 16769 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 183d364a616c4b37857e40285941660d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:06:37.239171 16771 catalog_manager.cc:1259] Loaded cluster ID: bd46ea81a61b44f5883f941f6dd2fb5e
I20260924 16:06:37.239215 16771 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 531228f7d6fe4c5b9124e2f2b2824802: loading cluster ID for follower catalog manager: success
I20260924 16:06:37.240687 16771 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 531228f7d6fe4c5b9124e2f2b2824802: acquiring CA information for follower catalog manager: success
I20260924 16:06:37.241642 16771 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 531228f7d6fe4c5b9124e2f2b2824802: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:06:37.440609 16857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:37.436883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43576 (local address 127.25.109.61:36745)
0924 16:06:37.437122 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:37.437126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:37.437191 (+ 65us) server_negotiation.cc:407] Connection header received
0924 16:06:37.437316 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:37.437319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:37.437444 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:37.437556 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:06:37.437799 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:37.438317 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:37.439119 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:37.439355 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:37.439453 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:37.439724 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:06:37.440181 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:06:37.440277 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:37.440336 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 16:06:37.440383 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":87,"threads_started":1}
I20260924 16:06:37.446885 16864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:37.443460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58864 (local address 127.25.109.60:39015)
0924 16:06:37.443715 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:37.443719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:37.443731 (+ 12us) server_negotiation.cc:407] Connection header received
0924 16:06:37.443812 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:37.443814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:37.443925 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:37.444021 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:06:37.444201 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:37.444788 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:37.445554 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:37.445774 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:37.445853 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:37.446094 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:06:37.446495 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:06:37.446599 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:37.446646 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 16:06:37.446690 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":65,"threads_started":1}
I20260924 16:06:37.454092 16857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:37.449755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43586 (local address 127.25.109.61:36745)
0924 16:06:37.449874 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:37.449881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:37.450412 (+ 531us) server_negotiation.cc:407] Connection header received
0924 16:06:37.450611 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:37.450614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:37.450659 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:37.450722 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:06:37.451037 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:37.451810 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:37.452880 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:37.453183 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:37.453291 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:37.453628 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:06:37.453692 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:06:37.453749 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:37.453892 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 16:06:37.453946 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 16:06:37.454361 16864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:37.450254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58880 (local address 127.25.109.60:39015)
0924 16:06:37.450392 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:37.450397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:37.450503 (+ 106us) server_negotiation.cc:407] Connection header received
0924 16:06:37.450610 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:37.450614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:37.450661 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:37.450733 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:06:37.451039 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:37.451512 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:37.452632 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:37.452951 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:37.453056 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:37.453302 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:06:37.453373 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:06:37.453449 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:37.454156 (+ 707us) server_negotiation.cc:299] Negotiation successful
0924 16:06:37.454210 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:06:37.455289 16871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:37.449731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46436 (local address 127.25.109.62:44327)
0924 16:06:37.449936 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:37.449940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:37.450108 (+ 168us) server_negotiation.cc:407] Connection header received
0924 16:06:37.450292 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:37.450296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:37.450467 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:37.450633 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:06:37.450987 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:37.451810 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:37.453480 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:37.453811 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:37.453912 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:37.454244 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:06:37.454606 (+ 362us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:06:37.454688 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:37.454841 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 16:06:37.455179 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":60,"threads_started":1}
I20260924 16:06:37.458318 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 16505
I20260924 16:06:37.465176 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 16582
I20260924 16:06:37.471191 26036 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCcKW0d/build/debug/bin/kudu with pid 16658
2026-09-24T16:06:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3101 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6157 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (288351 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST